9

I am currently creating an API and implemented a few basic REST actions such as get, update, search etc. Another program that collects data needs to synchronize these to this API. My current thought is: What would be good practice to synchronize this data?

A few informations: The amount of data that needs to be synchronized is different between applications. Some may transfer 100 records a minute, others 10k records a minute. There will be a transfer limit per hour to keep the server load balanced. Additionally, requests are signed with a token to check authenticity.

Here's a little pro/con list I've made just now. I'm neither sure if these are real pros/cons there nor am I sure if I should weight some of them.

Option One
Loop through this data and create an API call for every record.

Pros:

  • integrates better with the current REST API
  • failed synchronisation entries can be queued for a resend

Cons:

  • a bigger overhead because of multiple calls

Option Two
Build a package of all those data and send over that package in one call.

Pros:

  • less overhead
  • just one query

Cons:

  • transfer error may drop all data
  • separate implementation to allow
Improve this question
3
  • 1
    Is there a concern with disconnects during the transmission process or with duplicate data being sent in the batches?
    – rjzii
    Feb 9, 2012 at 16:28
  • I'm not concerned that there are foreseen disconnects. Also the infrastructure should be stable enough. The only concern is that all data need to arrive properly to generate statistics.
    – pdu
    Feb 9, 2012 at 16:58
  • 1
    Be careful about trusting the infrastructure, at a minimum you should assume that things will fail at some point in your code so that you don't leave things in an inconsistent state if the worst happens.
    – rjzii
    Feb 9, 2012 at 17:26

1 Answer 1

Reset to default
7

Generally, the overhead of initiating and ending the remote request is high enough that you are going to want to batch the calls. Exactly where this line is really depends on how much data is involved here to say where your batching lines should be. It really should not be that much overhead to make the receiving system handle batches -- computers are real good at "run this function again with new parameters" last time I checked.

From a logical perspective, would a batch be considered a transaction in the typical database sense? If so, it also makes loads of sense to let it travel and succeed (or fail) together rather than trying to work out how to spread a single logical transaction over separate physical transactions.

What would push me towards doing single calls is if each request had a lot of data involved -- hundreds of KB at least -- or if things needed to be a bit more real time and you couldn't wait to build batches.

Improve this answer
2
  • Single requests would be small. A batch of 500 records may be around 4mb. Considering that these are just strings, I could compress them before sending, reducing the file size. The only problem I'd have with sending a file to the api is signing parameters with the api token.
    – pdu
    Feb 9, 2012 at 17:05
  • 1
    Doesn't need to be a file -- just make the API take a collection of updates rather than a single update.
    – Wyatt Barnett
    Feb 9, 2012 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.