A separate endpoint specifically for sync in a REST api

Question

I am working on a multi client/single server app. Communication between the two happens via RESTful API. There are several resources that need to stay in sync across all clients, including images captured from camera, likes and comments on these images, user profiles and user followers/followings. My current approach involves GETing all resources changed on server since last sync, like GET /images?since=<last-sync-timestamp>, GET /users/:id/followers?since=<last-sync-timestamp> etc. and POSTing all client changes since last sync to the same endpoints, like POST /images, POST /user/:id/followers

The main drawback to this approach is way too many network calls - For each resource I need to sync, I need to perform at least 2 network calls.

So I am thinking about introducing a new endpoint like /sync which provides all syncable resources that have changed since last sync timestamp. Both client and server will need to parse the JSON and put the respective data in appropriate tables, but now, no matter how many resources I have to sync, I will only need to perform two network calls, but change the structure of the JSON as reosurces get added/removed.

Any thoughts on why the second approach is superior/inferior to the first one?

Answer 1

In REST, you should think of URIs as identifying an addressable resource. So as long as you think of a "sync" as a noun ;) you can define that resource in whatever manner suits your requirements.

At surface level, in REST we talk about collection resources (like the collection of users) and individual resources (like a single user), so a resource that is an aggregate of other resources is in keeping with that.

The idea of "a resource" is extremely broad. I don't know that there's a precise definition. So an aggregate resource that bundles up the smaller resources saves bandwidth and reduces latency sounds like a good idea.

Answer 2

Second approach is a bad/inferior idea.

• You lose the ability to cache individual resources using standard caching methods (both on the server and client).
• You lose the ability to sync individual resource state using HTTP headers and timestamps
• You lose the ability to update resources as you go (why wait to push a change to the server, just push it when user is ready to push it)
• You will have a catch all content type for every single resource you have, rather than one for each resource type (ie your JSON will have to incorporate all types of data
• Your client will be overly complex, instead of dealing with changing a resource and pushing it to server it has to handle the sync resource.

All to avoid making more small network calls. If you use HTTP headers and gzip compression these calls should be small to tiny, so what is big deal? You gain so much more than you lose.

Answer 3

How much difference in application performance or server overhead is having one network call rather than two per update going to make? Well, it depends on how many simultaneous clients you expect, and how often they'll be polling for updates. If you have a "small number of clients" like 10 or 50, and they request updates every N seconds (say N between 10 and 120), you can do the math and find it's not a substantial network or server load, so you'd probably be wasting your time optimizing it.

On the other hand, if you had 100s, 1Ks, or 1Ms of simultaneous clients, and/or they're requesting updates every second or few seconds, and/or you have not just two kinds of objects to communicate but M>2, by all means do everything possible to optimize the interactions.

This still leaves open a key question: Is adding new mega-REST endpoint (/sync) really the best way to do that?

Maybe not. Cormac Mulhall outlined some of the downsides. A more subtle problem is that sync is fundamentally hard. It seems not so difficult, but a lot of shops have repeatedly broken their picks on it.

Why is sync hard? Well, let's just scratch that surface. You mention URLs specifying ?since=<last-sync-timestamp>. Is that the server's timestamp, or the client's? They will be different. Distributed timekeeping is itself hard. Clocks vary. You can help this along by adding another rest call /time that helps the client estimate how far its clock is off the server's clock. But now you've got the complexity of time drift estimation, just like inside Network Time Protocol. REST isn't so simple any more. And just what does a timestamp mean, anyway? ?since=1418070480 means...? Is it the start of that second, or the end? Keep in mind that CPUs can execute ~10**11 instructions per second. Seconds are a crude metric. So you say: "You know what? There's going to be error, so I'll just put in a margin for error." You coordinate clocks as best you can, then add another 20 or 30 or 60 seconds back, just to catch stragglers. Now you've think you've got the "accidentally missing updates" problem licked--you haven't really, in a system of any complexity, but let's just pretend. But you've created another problem: The possibility of duplication, because you're asking for another N seconds of data before where you really think you need it. You'll have to build client-side code to make sure objects aren't double-posted. REST was supposed to be simple!

Even with all this, you haven't addressed the elephant in the room: You still have a polling-style update, where the clients have to constantly ask, ask, and re-ask: "Anything new? No? Okay. How about now? No? Okay. How about now? No? Okay...." Your server is still getting hit with needless requests. 90+% of which will always answer "Nothing new to report."

The truth is, you do not yet have a communication style that is even half-way suited to synchronization requirements. Polling is always going to be a chatty, server-loading technique. What you really need aren't AJAX or AJAJ update polls, but at a minimum Comet (long-polling reverse AJAX calls)--or even better, a full back-and-forth communication mechanism such as WebSockets. These allow your server-side code to establish push techniques or push-pull interactions that, while not automagically fixing all of the semantic difficulties of synchronization in a distributed world, substantially reduce the network call, server overhead, and code complexity required to have truly synchronized endpoints.

In summary, one mega endpoint rather than two individual ones is a minor issue. If you want efficient temporal synchronization, the real issue is you need a better interaction mechanism.

Answer 4

Serving smaller and more numerous requests should not be so much of a risk to the server, unless you leave the synchronization as a user-triggered event.

See, when you leave the synchronization to the user, it is always going to be heavy, regardless of whether the requests are small but many, or few but big, you're always at risk of having too much network traffic to handle - users tend to hit the refresh button a lot more than necessary, especially if the app seems slow or unresponsive to the refresh event (which means, as soon as you have a heavy load, it will get even heavier).

But there are other ways. And I don't mean to tell you to put a timer on your app, and poll for changes every x seconds. I mean using something truly efficient, like push notifications.

Push notifications can be used to tell the client app that there are changes to be synched, and optionally, which exact resources need to be synched. This technique is called send-to-sync.

With send-to-sync, when something is changed in the server (by receiving a POST/PUT operation), you can make the server notify all outdated clients that they have to synchronize with that change. You can even get rid of the timestamp in your requests, and leave things as simple as possible in terms of REST resources, and also completely avoid having to account for time differences in the many devices.

Answer 5

Here is how we do it , am not sure if it's good or bad but is works so good.

I too have an dedicated endpoint for sync

/sync/{syncKey}

As soon as the user authenticated , if the user client (mobile apps) want to have sync they can call add to sync API , ther I store one entry per device per user , and return the entry key as sync key

Then whenever any changes happened on any resource for that user I will add the entity key (only primary key) to all the devices sync entry for that user ( it supports all CRUD operations)

Then the client call the get sync API using the sync key to fetch the sync entry, if any key is present for any resources then client will make separate API call for each resource using the resource key and make appropriate changes in there end and finally send an clear sync API calls to remove the synced keys from db

When the user logs out client will send delete request to remove the sync entry completely