1

I am developing a web application for a client, connected to 4 differents sources :

  • E-commerce website
  • CRM
  • Emailing platform
  • Surveys platform

All the sources have REST Api's.

The web application has to get the data from the sources and provide a list of customers. Also the client can create/edit customers, so the web application have to send the new / updated customer to the sources.

On the sources, the client can also create/edit customers.

The 4 sources have the same columns (email, firstname, lastname, etc.)

The web application has to provide a unique list of customers (unique by email address).

On the web application and the source i have access to an "updatedAt" value.

This is the workflow i want to setup :

  • A crontab each 5 minutes > get last customers from source 1
  • Check on the web application if a more recent version exists
  • If not insert / update customer
  • Then send to the other sources the new / updated customer (the updatedAt value will change on each source)

But on the next cron execution (for source 2) the previous updated customers will be returned because of the updatedAt column, so i'm afraid, the last customers list will increase and increase..

I can't control the change of the updatedAt column on the sources.

I feel stuck!

My questions are :

  • What is the best way to handle this kind of system ?
  • Is my workflow idea in a good direction or totally wrong ?

Thanks in advance :)

1

You're on the right track, but you'll actually want to look into another mechanism called Webhooks to accomplish what you're looking for. This Article describes the basics but if they exist on the apis you're interacting with, it will remove the need for your workflow idea.

The basic pattern is that you have a subscriber on your end that will subscribe to a number of events. Once that subscriber is notified by the webhooks, He'll take an appropriate action. For example, let's say the Customer Information API source you're working with has a webhook for when customer info updates. Your workflow would look like this:

  • When a webhook fires, go to the API and get the updated version
  • Save this information to the matching record in my database

That's way simpler than yours.

HOWEVER: if these aren't available, your update workflow seems just fine. I'd go a step further though and for efficiency, only ask for information that's new since your last update. IE, if my last update was 3:00, i'm only interested in changes since 3:00 and now.

| improve this answer | |
  • Thank you for your response, unfortunately there is no webhooks on 3 sources (it could be possible to add a kind of webhooks on the e-commerce website, but i'll have to spend more time on the project, and that is not possible). So I think i will continue on the workflow i described. Did you see my remark on the second cron execution and the potential problem with the updatedAt column ? – Benjamin B. Feb 22 '19 at 22:25
  • I did, yeah. I suppose that i wasn't clear. I meant that for each source to restrict for recrods updated in the last 5 minutes. – Adam Wells Feb 22 '19 at 22:29
  • 00h00 => Get contact from source #1 updatedAt >23h55 (found one 23h56)=> Send to the other sources (change updatedAt everywhere to 23h56) 00h05 => Get contact from source #2 updatedAt > 00h00 , etc. – Benjamin B. Feb 22 '19 at 22:35
  • I think i will have a problem because if i sync the other sources at 00h00, they will have the updatedAt = 00h00, and the next cron will take them, etc, what do you think about that ? – Benjamin B. Feb 22 '19 at 22:41
0

I think you have a design problem that will lead to inconsistency in your data sources because none of them seems to own the "truth". What happens if I update the email in one of them and update the name in the other? Determining that one of them is the entire truth is very hard to define.

There are different solutions to this problem depending on how much you "own" the code of each component:

Have a single source of truth

If you can modify the code in all of them the have one of them own the data and the others just be pass through APIs that ultimately just call the "truth" API so that all the updates are one way and you dont need to sync data. This also creates a single point of failure for the functionality so be aware that this could couple your system in bad ways (there ways around it tho... Like maybe have a shared cache?)

Keep the current design but add pre-write validation

Before any system attempts to "write" a new version of the data, check with the owner of the data if the version the user is seeing is the latest, if it's not and the update would cause a conflict then reject it and return the latest version to the user with a nice error message

Clown factory

When syncing perform a "merge" instead of a replace of the record so you only change the pieces of data that were modified when segregating the changes across the other systems. Keep a log of the changes somewhere (hopefully in the order that they happened) so you can trace them without losing data in case things go wrong and you need to go back a few versions to recover unintended overwritten data.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.