How do I ensure consistency between two separate systems which store and maintain the same set of data, when the method of communication between them is REST APIs and webhooks? Is it possible?
I find myself coming across this problem when working with APIs like MailChimp or eBay, where the third party system is acting as a complimentary system to an internal CRM or an online shop. (Although I'm mentioning specific services here, what I'm getting at is a general way of achieving consistency using APIs and webhooks, which is why I think the question belongs here.)
Let's say I call my application MyApp, and the third party application ThirdApp. Both store information on my clients. A unique ID, an email address, and some preferences.
When a client is updated on MyApp, I can make an API call to ThirdApp to inform them of the changes, so they can update their version of the data.
Similarly, if someone makes a change to a client entity on ThirdApp, I can have a webhook set up so that ThirdApp can inform MyApp of the change.
I don't want to be waiting for API calls to complete each time I update a client on MyApp - if ThirdApp's API stops responding, then MyApp would be unable to make changes, which is undesirable.
Similarly, ThirdApp just assumes MyApp is okay with any changes it applies, anyway. It doesn't wait to commit it's changes - the webhook request it sends to advise MyApp of the change is really just a courtesy message.
Is it possible to ensure (perhaps eventual?) consistency in this situation?
Maybe a change queuing system?
I've thought about using some sort of intermediary queuing system on the MyApp side, with incoming and outgoing changes, but I always seem to be able to think of ways things might end up inconsistent. For example, imagine if client
123 has an address of
email@example.com, and the pending queue looks something like this (none of the queue entries have been processed yet):
- OUTGOING MyApp says client ID
123's email address has changed to
- INCOMING ThirdApp says client ID
123's email address has changed to
Whether the queue has been processed or not, ThirdApp is correct in storing
firstname.lastname@example.org as the client's email address. However, as we sequentially process the queue:
- The outgoing change sends an API call to ThirdApp, "the new email address for
- ... some other stuff happens, during which time both email addresses are wrong...
- The incoming change means MyApp updates it's
123entity so that the email address is
email@example.com. MyApp is now correct, but ThirdApp is wrong.
Possibly to solve that I could implement some sort of look-ahead logic in the queue processor, which will try and determine what the "correct" action should be, based on what other subsequent entries affect the given record further down the queue. This seems like it could be very complicated though.
Maybe a third, single source of truth?
Maybe the solution is to have a third copy of the information, managed by a third system, which both MyApp and ThirdApp send update information to, and receives updates from. So any change, whether from MyApp or ThirdApp, is an incoming change there, and outgoing API calls are made to update other keepers of the data accordingly.
At this point I'm starting to really struggle to keep track of consistency in my head. Also, I'm sceptical that adding another copy of the data into the mix will solve the problem - it seems likely that it would make it more complex, and therefore worse.
I also find myself wondering whether I can trust a webhook request from ThirdApp to be timely or not. What if the change was made 20 minutes ago, but the webhook request only just got sent? Changes within the last 20 minutes therefore should supersede them.