0

Hypothetical scenario:

Let's say I have a Customer service. It has 1 million customers in its database. I then create a new service that depends on a specific projection of the customer entity.

I want to avoid coupling and synchronous communication, so the new service should get customer updates through asynchronous messages.

But how do I sync the existing 1 million customers to the new service?

I've read suggestions that the Customer service should simply emit messages for all the customer entities, but there are a couple of problems with that solution:

  1. The new service needs all the data from the start. Failure because a needed customer isn't synced yet it not acceptable.
  2. Consuming 1 million messages is slow. I admit that I might have a suboptimal implementation here, but having Azure Functions trigger on messages and write updates to Azure Sql Database will quickly consume connections and hit the limit, unless properly throttled. Throttling slows the sync down and exacerbates point 1.

Is there a better way?

I'm thinking some kind of tool that writes directly from database A to database B, while mapping the model from A to B.

General suggestions and suggestions specific for the Azure context are appreciated!

Edit: what if the new service needs to sync data from two or more services?

4
  • 1
    backup + restore? Oct 19, 2021 at 21:38
  • Yeah I guess that could work! Then change to desired schema after the fact. Or maybe one could change schema before restore?
    – AleksG
    Oct 19, 2021 at 21:50
  • 2
    For this you should probably do a database migration, which is a broad topic unto itself. If you are .NET then I suggest Data Migration Assistant as a starting point.
    – John Wu
    Oct 19, 2021 at 22:42
  • @JohnWu Thanks for the hint, this looks promising.
    – AleksG
    Oct 20, 2021 at 6:42

1 Answer 1

2

If this is a one-off process, a backup/restore type process is probably the best way to go. Personally, I would suck it up and go with the RabbitMQ message route myself. Its less prone to mis-matched schema problems I would imagine

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.