0

We are replacing a legacy application that runs on a single Oracle DB, and the decision has been made to replace it with a microservices-based application running on Azure SQL.

Do we do data migration from Oracle via the microservices, or do we do direct data migration from Oracle to Azure SQL DBs?

Advantage I see with doing it via the microservices, is that the business logic in the microservices takes care of data transformation and load to the target DBs. Whereas if you do direct migration to the target DBs, then you have to develop separate transformation and load processes.

Disadvantages with using the microservices for data migration:

  • you need to talk to the microservices via their API - so you'd have to extract data from Oracle, then convert it to a format that's compatible with the API (eg JSON). I don't know if that would be fairly easy or extremely hard to do.
  • you're dependent on microservices being built before you can run or test data migration. ie, you can't run or test data migration processes independently of the development of business logic.

Other questons: maybe migrating data via microservices is the only option if there are complications relating to how the relevant data can be persisted. Eg, the new system may say that you can't create a customer without an address, whereas in the old system, this was allowed, so you may have source address records that aren't related to a customer, which is valid in the old system but won't be allowed in the new system.

I'm very much leaning towards building separate transformation and load processes for the Oracle data, as it just seems a lot easier than trying to 'feed' all the data in the right format to a very large no. of microservices.

I should also say that I'm thinking that we'll migrate from Oracle to Azure SQL DBs using SQL Server Migration Assistant, wherever possible. But the desing of the new data model/data schema is very much 'up in the air' at this stage, so not sure to what extent we'll be able to use SSMA.

On the other hand, maybe it just doesn't make any sense NOT to do the data migration via microservices. Because it's only when the granularity of the microservices is finalised that you know exactly how many DBs you'll have and what data they'll store. So, it's only at this point that it makes sense to start building and testing data migration?

Edit: thanks v much for the reply, that recommends either of 2 approaches: historic data store (HDS) or ‘clone and cut’.

• So, with HDS, each microservice has 2 DBs: the HDS (which is a copy of the Oracle DB, migrated to 1 Azure SQL DB) and its own, separate Azure SQL DB. Each microservice needs to get the ‘historic’ data it needs from the HDS, and this could be done either on-demand, or all the data it needs could be extracted from the HDS and loaded to the ‘new’ Azure SQL DB in 1 ‘hit’. So, each microservice would need to ‘understand’ how to ‘talk’ to both the HDS and its ‘new’ Azure SQL DB – the structure of the data in each may be very different, so isn’t this hard to do? Also, presumably, the general logic would need to be: if a user wants to read/update/delete an existing record, then first look for it in the HDS and if it isn’t there, then look for it in the ‘new’ Azure SQL DB? (this would apply if the microservice queries the HDS on-demand.)

• ‘Clone and cut’ means that each microservice has 1 Azure SQL DB, which would initially be a copy of the Oracle source DB. As the microservices are developed, you keep and remove the DB entities, as appropriate, for each microservice. There is the issue of what happens where those developing different microservices decide that their microservices both need the same table. In these cases, there needs to be coordination to decide which microservice has direct access to the relevant table, and which microservice/s will need to access that table via another microservice. You seem to assume that the source Oracle DB can be ‘cloned’ as many Azure SQL DBs (1 for each microservice), and that no further data migration is needed once this happens. However, the legacy system will be (and needs to) be kept running until cut-over to the new system. I expect that development of the microservices will occur over many months – so the basic question is, how do you migrate all the data that is created/updated/deleted between the ‘cloning’ and cut-over?

  • Sounds like you're pretty clear about the advantages and disadvantages of each approach. What prevents you from making a decision? – Robert Harvey Apr 10 at 16:35
  • hi, basically, I'm not sure whether the advantages and disadvantages I mentioned in my OP are valid...or, even whether the approach of building data migration processes that are different and separate from microservices makes sense... – Roger Mouton Apr 11 at 1:52
0

Clone and cut

Give every team an exact duplicate of the database. It is their job to prune and transform this data as fits their needs.

They can achieve this transformation via their own database management process. They can fill in the blanks, cross the t's, and dot their i's.

Of course you may want to keep a copy of the base data around until everything has settled down. Nothing worse than that woops moment when a table is deleted but was actually really important.

Historic Data Service

Instead of parcelling out the data to the new services, provide it as an historic data store.

Each new micro service can access this historic data, and handle its nuances as needed.

Some micro services might pull this data on demand due to a query, or they might perform a single batched dump and load, subsequently forgetting that the service ever existed.

You may choose to decommissioned the historic data service later, but hopefully now the new micro service landscape is healthy and well understood, allowing that data to be backfilled, or discarded case by case.

Migrate to Azure SQL

Either way migrate the oracle db as a direct copy into Azure first. This then gives you replication, and duplication tools. It also ensures that everyone is dealing with the same data store technology reducing the chances of a oops.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.