Synchronize multiples REST api, centralized into 1 web application

Question

I am developing a web application for a client, connected to 4 differents sources :

• E-commerce website
• CRM
• Emailing platform
• Surveys platform

All the sources have REST Api's.

The web application has to get the data from the sources and provide a list of customers. Also the client can create/edit customers, so the web application have to send the new / updated customer to the sources.

On the sources, the client can also create/edit customers.

The 4 sources have the same columns (email, firstname, lastname, etc.)

The web application has to provide a unique list of customers (unique by email address).

On the web application and the source i have access to an "updatedAt" value.

This is the workflow i want to setup :

• A crontab each 5 minutes > get last customers from source 1
• Check on the web application if a more recent version exists
• If not insert / update customer
• Then send to the other sources the new / updated customer (the updatedAt value will change on each source)

But on the next cron execution (for source 2) the previous updated customers will be returned because of the updatedAt column, so i'm afraid, the last customers list will increase and increase..

I can't control the change of the updatedAt column on the sources.

I feel stuck!

My questions are :

• What is the best way to handle this kind of system ?
• Is my workflow idea in a good direction or totally wrong ?

Thanks in advance :)

Answer 1

You're on the right track, but you'll actually want to look into another mechanism called Webhooks to accomplish what you're looking for. This Article describes the basics but if they exist on the apis you're interacting with, it will remove the need for your workflow idea.

The basic pattern is that you have a subscriber on your end that will subscribe to a number of events. Once that subscriber is notified by the webhooks, He'll take an appropriate action. For example, let's say the Customer Information API source you're working with has a webhook for when customer info updates. Your workflow would look like this:

• When a webhook fires, go to the API and get the updated version
• Save this information to the matching record in my database

That's way simpler than yours.

HOWEVER: if these aren't available, your update workflow seems just fine. I'd go a step further though and for efficiency, only ask for information that's new since your last update. IE, if my last update was 3:00, i'm only interested in changes since 3:00 and now.

Answer 2

I think you have a design problem that will lead to inconsistency in your data sources because none of them seems to own the "truth". What happens if I update the email in one of them and update the name in the other? Determining that one of them is the entire truth is very hard to define.

There are different solutions to this problem depending on how much you "own" the code of each component:

Have a single source of truth

If you can modify the code in all of them the have one of them own the data and the others just be pass through APIs that ultimately just call the "truth" API so that all the updates are one way and you dont need to sync data. This also creates a single point of failure for the functionality so be aware that this could couple your system in bad ways (there ways around it tho... Like maybe have a shared cache?)

Keep the current design but add pre-write validation

Before any system attempts to "write" a new version of the data, check with the owner of the data if the version the user is seeing is the latest, if it's not and the update would cause a conflict then reject it and return the latest version to the user with a nice error message

Clown factory

When syncing perform a "merge" instead of a replace of the record so you only change the pieces of data that were modified when segregating the changes across the other systems. Keep a log of the changes somewhere (hopefully in the order that they happened) so you can trace them without losing data in case things go wrong and you need to go back a few versions to recover unintended overwritten data.