I'm building an application that queries several APIs and duplicates some of the data in a downstream, centralized database to make it easier to perform some processing on it. My application will only read these APIs.

Checking for new and updated content seems easy enough (I can sort the data by last updated/modified). I'm having difficulty conceptualizing how to track deleted data. These APIs do not notify you when data is deleted; the record simply doesn't exist any more.

Is there a pattern or best practice I can look at? I could brute-force it, i.e. check to make sure each piece of data still exists on the server at some specified interval, but that seems pretty intensive.

I have tried looking solutions online, but was unable to find anything useful.

  • One method I used to determine if records were deleted was to keep the data from the previous run and compare it with the data from the current run. I collected all records each run. Any records on the previous run that weren't on the current run were considered "deleted". – Gilbert Le Blanc Apr 11 '16 at 17:25
  • Are you filtering the data you're downloading or are you downloading all of it and then sorting in your app? – JeffO Apr 11 '16 at 19:50
  • @jeffO I had been planning to drop a datetime value in the db to record the last time a check was performed for a specific api. Then, I would query the api for all modified (and new) records after the last checked date to reduce the amount of data transferred. This will work well, I think, for new and updated records, but leaves me with no way to know if a record was deleted. – user81971 Apr 12 '16 at 12:08

If your data source does not provide you with notifications when data is deleted, then I don't see any other choice but to brute-force the solution by polling the source periodically.

Depending on the data source, this may violate terms of service if you poll too often because it puts a heavy load on the source server.

The ideal solution is to work with the owners of the APIs and convince them to add support for tracking inserts updates and deletes (e.g., "list all recordIds that were deleted in the last 24 hours"). Sadly, this is not always possible.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.