0

Scenario/Context

There is an old monolithic desktop application (running locally) which uses ODBC to connect to SQL database (on premise). This desktop application should be replaced with a new web application (microservices in the public cloud). As the application offers a considerably large amount of functionality, the strategy should by to migrate one part after another into the cloud instead of having a hard switch after several years of development. Therefore the need comes up to run both systems in parallel and let them sync their data.

Question

What is the best/cleanest way of achieving a data synchronization between both system? Especially considering that they will likely have different database structures and database systems. The goal should be that in case of an entity being created in the old system, it is mapped and then created in the new system and vice versa.

My thoughts

There are migration patterns that might be applicable like the the anti corruption layer (ACL) and strangler-fig pattern. Synchronizing from the new cloud application to the on premise database seems from my perspective fairly simple using the ACL pattern. But the other way around it looks a bit more tricky: The strangler-fig pattern seems to be more applicable on migrating a monolithic web application backend in which case I could simply intercept the HTTP requests using some kind of gateway and route based on path/functionality. Applying that to this case seems tricky, the only way I could imagine is proxying the ODBC traffic, but that seems quite complex to me. Therefore I was thinking of another event interceptor to grab the data: change data capture (CDC). Basically configuring the database to capture the changes and then regularly reading these changes from the new system/ACL. The issue with calling the ACL directly from the old desktop application is from my perspective the risk of inconsistency. The user could simply shut down the PC during a transaction to the on premise db and ACL leaving one of them inconsistent with the other.

2 Answers 2

2

Frankly the proposal sounds like a recipe for disaster.

If you have a perfectly working application for the time being, and you have several years to complete a transition "to the cloud", then why embark on the difficulties of constantly synchronising two different storage schemas and two front ends?

You'll almost certainly either introduce serious faults, or massively increase analysis effort and reduce your pace of progress to a crawl (relative to whatever the pace would be if designing the new system independently and having a one-time migration of data).

And you're eventually conserving nothing of the old system. You're not keeping the front-end, and you're not keeping the back-end and storage schema. Yet the old will act as a dead hand on the design of the new application, because the two have to be fully compatible during the transitional period.

I'd seriously re-examine the rationale of the proposal.

I imagine the perceived benefit is that the project seems incremental, and that there is some ongoing acceptance testing of the new interface, and some demonstrable progress which seems useful to the business.

But the disadvantages will be severely higher development complexity and testing requirements, reduced design freedom for the new system, and a long period of years during which the development team will have this monstrous chimerical whole to deal with, the totality of which will be vastly more complicated to oversee than either the old or new individually.

My advice would be different if you were just making a different web-based shell for the same back-end, and shifting that backend to the cloud (i.e. from hardware you own to hardware you don't).

You might also find that the apparent incrementalism of your proposed approach, is allowing you to avoid recognising the sheer scale of what you're embarking upon and the degree of risk.

If the existing application took several years to produce in its current form, I'll bet it takes at least that amount of time to produce a new design, and multiply that figure further to account for the transitional development which will be required to make the old and new systems work together whilst both are online.

Companies which actually execute these kinds of transitional developments have powerful resources at their disposal, and equally powerful business justifications - their project leader wouldn't be asking for advice on SE, but would be hiring a panel of paid experts, because the money would be a drop in the ocean of their budget. Think about where your own resources fit in the scheme of things.

1
  • Thanks for your extensive answer/advice on this, Steve. I guess it is a mixture of the management who asks for fast results and the current desktop app being far from ideal and perfectly working as well, which led to the ask for this run both in parallel approach from the customer. I will reevaluate this considering our resources.
    – Malte G
    Feb 13 at 13:54
2

Especially considering that they will likely have different database structures and database systems

Do not do this. They have (almost?) identical functionality, so they should have identical database structures. You should develop the new application as a on-prem web application running against the SAME DATABASE as the existing system. (Well, in final deployment - you'll also want a test DB and staging DB, but perhaps the existing application also has those).

Once the shift to the new system is complete and the old ODBC access turned off, you can "lift and shift" the DB and application into the cloud, and maybe start thinking about partitioning the DB if required.

Anything else will result in horrific data loss incidents when you least expect it.

4
  • The database design evolved within the last almost 20 years. It is actually one of the bottle necks in the current application. But I guess this is something that might be fixable before starting with a migration. Nevertheless the plan is to create a microservice application, so ideally you would have a database per microservice. Would you postpone this step then, or how would you approach this?
    – Malte G
    Feb 13 at 14:02
  • 1
    As usual: why are you creating a microservice application? How many millions of users do you have?
    – pjc50
    Feb 13 at 14:37
  • 1
    In order to split the database across microservices, without losing data, you will first have to get a big chart of all the tables and queries, and then work out where you can sensibly cut across it. Every JOIN you split across microservices will have a negative performance impact.
    – pjc50
    Feb 13 at 14:38
  • @MalteG, the existing database design may not be great and may benefit from change, but for OLTP loads you should invariably start from the principle of one database per company if possible, and certainly no more than one database per major OLTP application.
    – Steve
    Feb 13 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.