I am developing a web application for a client, connected to 4 differents sources :

  • E-commerce website
  • CRM
  • Emailing platform
  • Surveys platform

All the sources have REST Api's.

The web application has to get the data from the sources and provide a list of customers. Also the client can create/edit customers, so the web application have to send the new / updated customer to the sources.

On the sources, the client can also create/edit customers.

The 4 sources have the same columns (email, firstname, lastname, etc.)

The web application has to provide a unique list of customers (unique by email address).

On the web application and the source i have access to an "updatedAt" value.

This is the workflow i want to setup :

  • A crontab each 5 minutes > get last customers from source 1
  • Check on the web application if a more recent version exists
  • If not insert / update customer
  • Then send to the other sources the new / updated customer (the updatedAt value will change on each source)

But on the next cron execution (for source 2) the previous updated customers will be returned because of the updatedAt column, so i'm afraid, the last customers list will increase and increase..

I can't control the change of the updatedAt column on the sources.

I feel stuck!

My questions are :

  • What is the best way to handle this kind of system ?
  • Is my workflow idea in a good direction or totally wrong ?

Thanks in advance :)


You're on the right track, but you'll actually want to look into another mechanism called Webhooks to accomplish what you're looking for. This Article describes the basics but if they exist on the apis you're interacting with, it will remove the need for your workflow idea.

The basic pattern is that you have a subscriber on your end that will subscribe to a number of events. Once that subscriber is notified by the webhooks, He'll take an appropriate action. For example, let's say the Customer Information API source you're working with has a webhook for when customer info updates. Your workflow would look like this:

  • When a webhook fires, go to the API and get the updated version
  • Save this information to the matching record in my database

That's way simpler than yours.

HOWEVER: if these aren't available, your update workflow seems just fine. I'd go a step further though and for efficiency, only ask for information that's new since your last update. IE, if my last update was 3:00, i'm only interested in changes since 3:00 and now.

  • Thank you for your response, unfortunately there is no webhooks on 3 sources (it could be possible to add a kind of webhooks on the e-commerce website, but i'll have to spend more time on the project, and that is not possible). So I think i will continue on the workflow i described. Did you see my remark on the second cron execution and the potential problem with the updatedAt column ? – Benjamin B. Feb 22 '19 at 22:25
  • I did, yeah. I suppose that i wasn't clear. I meant that for each source to restrict for recrods updated in the last 5 minutes. – Adam Wells Feb 22 '19 at 22:29
  • 00h00 => Get contact from source #1 updatedAt >23h55 (found one 23h56)=> Send to the other sources (change updatedAt everywhere to 23h56) 00h05 => Get contact from source #2 updatedAt > 00h00 , etc. – Benjamin B. Feb 22 '19 at 22:35
  • I think i will have a problem because if i sync the other sources at 00h00, they will have the updatedAt = 00h00, and the next cron will take them, etc, what do you think about that ? – Benjamin B. Feb 22 '19 at 22:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.