0

I have a web application and a REST API server (Microsoft Dynamics CRM 2016) that I don't have access to it's code, but I have access to the available resources. I need to create a data sync between both servers, some of the tables are one way and the others two way data sync has to be done, I haven't found any good solution to solve the problem.

I can handle the one way data sync by creating a log table of each SQL statement executed in my web application, and after a period of time I can use the REST API endpoints to sync data from my web application into the REST API server, but how to do the inverse (i.e sync data from REST API application into my web application)?

The naive solution are to get all table records from REST API server and then start syncing by comparing records from both sources, given that I currently don't have timestamp on the records, and I have no way to get the resources that are modified/inserted from the REST Server, because of that I should get all the resources from REST server and not the changed records which will increase the network traffic during every sync operation.

Insertion are not a problem, the problem is with update and delete operations.

Improve this question
1
  • Look into Dynamics 365 change tracking which will allow you to query only for records updated since the last time you checked.
    – Matt
    Jun 28, 2017 at 12:25

1 Answer 1

Reset to default
0

Given the scenario where you can't control/access REST server, you have to make calls for every table until REST service expose some API to enable this feature. But off-course you can optimize data flowing through network:

  1. Create list of tables which have updates more recent than your service (by querying REST service).
  2. Now only sync tables from above list.

or,

Try to fetch updated/new data in one call (in step 1).

In my opinion above solutions are not fundamentally different from your current solution but they do try to control amount of traffic in network. You have to implement both and run load test on them to see what fits best.

Improve this answer
11
  • step 1 is too abstract could you explain please?
    – M.Abulsoud
    Aug 28, 2016 at 1:46
  • @M.Abulsoud step 1 comes from same step as you are currently doing. syncing up tables at your side and on REST side. instead of full table sync just fire queries like are there any updates on table T1 since you last synced. in this way you will be able to reduce traffic.
    – Prateek Jain
    Aug 29, 2016 at 15:41
  • ok, I handle what you said by create a flag in the history table called pending if the record not synced for x value in time then it's value 'Y', and after finishing the syc I will change it's value to 'N' so that I will reduce the sync traffic from my side (number of requests going to the API server). Actually it's not hard for insert to handle the two way sync, but for update I cannot find a solution for it since the rest api is using odata protocol to represent the resources, I don't have a timestamp attribute in the resources but some of the resources has ETAG attribute.
    – M.Abulsoud
    Aug 30, 2016 at 6:37
  • Do you know how to compare records to check what record must win in update operation if I don't have the timestamp is that possible?
    – M.Abulsoud
    Aug 30, 2016 at 6:37
  • If you dont have information like insert or updated timestamp, this would be difficult to implement. Are you taking full data back up from REST, currently?
    – Prateek Jain
    Aug 30, 2016 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.