Synchronize multiples REST api, centralized into 1 web application

Question

I am developing a web application for a client, connected to 4 differents sources :

• E-commerce website
• CRM
• Emailing platform
• Surveys platform

All the sources have REST Api's.

The web application has to get the data from the sources and provide a list of customers. Also the client can create/edit customers, so the web application have to send the new / updated customer to the sources.

On the sources, the client can also create/edit customers.

The 4 sources have the same columns (email, firstname, lastname, etc.)

The web application has to provide a unique list of customers (unique by email address).

On the web application and the source i have access to an "updatedAt" value.

This is the workflow i want to setup :

• A crontab each 5 minutes > get last customers from source 1
• Check on the web application if a more recent version exists
• If not insert / update customer
• Then send to the other sources the new / updated customer (the updatedAt value will change on each source)

But on the next cron execution (for source 2) the previous updated customers will be returned because of the updatedAt column, so i'm afraid, the last customers list will increase and increase..

I can't control the change of the updatedAt column on the sources.

I feel stuck!

My questions are :

• What is the best way to handle this kind of system ?
• Is my workflow idea in a good direction or totally wrong ?

Thanks in advance :)

Answer 1

You're on the right track, but you'll actually want to look into another mechanism called Webhooks to accomplish what you're looking for. This Article describes the basics but if they exist on the apis you're interacting with, it will remove the need for your workflow idea.

The basic pattern is that you have a subscriber on your end that will subscribe to a number of events. Once that subscriber is notified by the webhooks, He'll take an appropriate action. For example, let's say the Customer Information API source you're working with has a webhook for when customer info updates. Your workflow would look like this:

• When a webhook fires, go to the API and get the updated version
• Save this information to the matching record in my database

That's way simpler than yours.

HOWEVER: if these aren't available, your update workflow seems just fine. I'd go a step further though and for efficiency, only ask for information that's new since your last update. IE, if my last update was 3:00, i'm only interested in changes since 3:00 and now.