6

I'm currently assigned a task, where I have to program a bidirectional synchronisation between 2 ERP-related software, one providing webservices (Visual Studio Web Reference) and the other one providing a REST API.

I managed to match the fields of the APIs together, so I could potentially start creating objects in both APIs, using the same data. However, I reached the point of the actual sync, and it's harder than I thought.

In my head, everything goes fine.

Grabbing the data -> determine, what change it is on which side -> sync the data from the source system to the target system

But in actual coding, I'm encountering some serious problems:

  • Both APIs don't have any way to tell, when data was modified the last time, making it hard to determine, what kind of change it is (I thought about having a list of the previous sync to compare, but that leads to a different problem)
  • There's a priority on the system with the REST API, so if there are 2 changes at the same time, the change in the REST API would count
  • In which order should they be sync'ed? If there's a focus on the REST API, then the REST API would need to apply their changes after the webservice one, right?
  • The sync software would run on a server, at the moment in a permanent loop of getting the data, comparing and applying the changes. I'd love to only sync the changed data, but that wouldn't be possible, regarding the problem, that there's no way for me to determine, what change occured (something was created, something was changed, something was deleted)

Can you give me advice on how to accomplish this, in a reliable and resource-friendly way?

The webservice API is a lot slower than the REST API, meaning there's an interval, where nothing gets sync'ed (it may sound stupid, but I thought I could solve this problem single-threaded). Could I potentially need multi-threading here? The webservice API is rate limited to 10000/hr, dunno if this could get a problem.

Improve this question
19
  • I would start by asking the person who assigned you the task if it is OK to let the last change win. There's really no such thing as "2 changes at the same time," and if there is, simply apply the REST change last.
    – Robert Harvey
    Aug 17, 2016 at 13:50
  • I'll update my post accordingly: The webservice API is a lot slower, which means, that there's an interval of not syncing. In this interval, there could be changes.
    – Jean Luc Nürrenberg
    Aug 17, 2016 at 13:51
  • Do these changes really have to happen in real-time, or merely whenever the system gets around to making them?
    – Robert Harvey
    Aug 17, 2016 at 13:55
  • No, they don't. They could also be applied like each few hours.
    – Jean Luc Nürrenberg
    Aug 17, 2016 at 13:56
  • 1
    @JeanLucNürrenberg You've only give us a keyhole to look through but based on what you've provided, I think the larger design is untenable. If you have two systems that are both primary sources for the same records, and either can be updated at either time, there's no real way to know which system has the correct current data. To really solve this issue, you need to address how the changes are being made. For example, you would build a service that takes the changes and applies them to both systems concurrently.
    – JimmyJames
    Aug 17, 2016 at 14:21

2 Answers 2

Reset to default
5

Welcome to the world of integration. The hardest thing about it is how poorly most vendors understand what you need and the resulting gaps in their interfaces.

In this case you seem to have a really intractable problem:

  1. You need to know precisely when changes were made
  2. There is no way to tell exactly when changes were made

No amount of design in your integration will fix this flaw in the API. You either have to get the vendors to address this issue (last update time, among other things should be included, there's no justification for not providing it) or you need to come up with a different approach to meet your requirements.

I think what you might be trying to do is on an interval, look at the two systems and apply any changes on one to the other. If both systems have changed in the interval you want to prefer on over the other. This is possible but you will probably lose changes made on the non-preferred system if your interval is fairly long. The way you can do this based on what you have provided is to look at all the records and compare them to some sort of record that you keep to determine if they have changed. For example you can keep a table of the record ids and a hash of the record. When the hash changes, you know there was a change made since the last time you checked. Then you can apply the update. This is not efficient and you can have a dirty read issue.

The problem you are trying to solve is essentially that of eventual consistency. Bitcoin, for example has a pretty interesting way of addressing it. I think if you read up on how it is accomplished, you'll find that you lack the requisite data elements to make it work.

Improve this answer
6
  • I can't remember the last time I worked in a system where this sort of thing mattered. Gaming systems, maybe?
    – Robert Harvey
    Aug 17, 2016 at 13:56
  • @RobertHarvey So if you deposited money into an account and it disappeared because of some timing issue around synchronization, that wouldn't matter to you?
    – JimmyJames
    Aug 17, 2016 at 14:02
  • That's a transactional problem, not a timing problem. You know that.
    – Robert Harvey
    Aug 17, 2016 at 14:03
  • 1
    @RobertHarvey I think your are missing the problem here. There are two systems that can have changes and changes in one system need to be reflected in the other. It's peer-to-peer, not client-server.
    – JimmyJames
    Aug 17, 2016 at 14:06
  • 1
    @RobertHarvey based on the description of the problem he is running into. It's a familiar scenario.
    – JimmyJames
    Aug 17, 2016 at 14:15
1

You can compare the data from both sites, but among others, I think you'll run into the reappearing deletion problem.

I think you'll need to maintain snapshots for comparison of one system with itself to find later what has changed within each system alone, then settle those changes with the other side while also updating the snapshot of each.

Using that kind of snapshot, you may miss some intermediate updates (e.g. same record updated twice), though I'm guessing that's probably ok.

However, I'd make sure the snapshots are atomically obtained, so the snapshot is fully correct for a single point in time.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.