2

I have a REST endpoint that gets a list of customers in a JSON format.

GET /webservice/customers

{
    "count" : 80267
    "customers" : [
        {
            "id": 1,
            "name": "John Doe",
            "email" : "[email protected]"
        },
        {
            "id": 2,
            "name": "Michael Bay",
            "email" : "[email protected]"
        }
    ]
}

My client saves in a database the customers it loads, but to update this list it needs to load all the data everytime. Sometimes I can have more than 100 000 entities, which is really slow and consuming.

I'm looking for a way to get only the data that have changed since the last update.

I found some answers like the If-Modified-Since or the Etag system, but if a new update is needed, the webservice stills getting all the data.

I thought about a system where I send the timestamp of the last update in a header or a URL parameter, and the REST endpoint only returns the new or updated entities, and the ids of entities that have been deleted since the last update.

But I suppose that a lot of clever people had this issue before me and I'm wondering if there is a common method to solve this issue.

PS : I do not need paging as the data is used to count customers after applying some filters on the dataset.

Improve this question
4
  • 1
    I do not need paging as the data is used to count customers after applying some filters on the dataset why don't you just return the count? /webservices/customers?filters=x&attribute=count
    – Laiv
    Jul 17, 2017 at 19:25
  • It's for a mobile app, the filters are changing often and the cost of a request is too high. Moreover I need to be able to do it offline
    – maxime
    Jul 17, 2017 at 22:07
  • Then you are more or less dumping the DB into the phone. Not even deltas will help you out here. The data growth factor matters more than any rest filter because you might retrieve 10 rows now and 100K rows ten minutes later. Not a big deal.
    – Laiv
    Jul 18, 2017 at 7:51
  • Indeed, I have an average of 500 updates (creations, updates and deletions) in a day. In a db of 100 000 rows, syncing 500 updates stays more efficient than recreating 100 000 ones. But yes, I'm almost syncing my database with webservice.
    – maxime
    Jul 18, 2017 at 8:03

1 Answer 1

Reset to default
7

Yes, you want to support filtering the endpoint by a date. Something like

GET /webservice/customers?since=XXX

If-Modified-Since and If-None-Match headers deal with returning or not returning a resource at all. You want to filter the responses from your endpoint.

Improve this answer
2
  • 1
    Good strategy. Also: Add a timestamp field to your returned object, parallel to count. That way clients can reliably know the last instant the server thought it provided data, rather than depending on its own clock. Distributed clocks can drift significantly. Having since-timestamp requests be off by even 1 second risks updates "falling through the cracks."
    – Jonathan Eunice
    Jul 17, 2017 at 22:14
  • I have the feeling that this solution is not going to work as expected. But Yes, filtering by last update date is almost a must.
    – Laiv
    Jul 17, 2017 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.