I'm building an application that queries several APIs and duplicates some of the data in a downstream, centralized database to make it easier to perform some processing on it. My application will only read these APIs.
Checking for new and updated content seems easy enough (I can sort the data by last updated/modified). I'm having difficulty conceptualizing how to track deleted data. These APIs do not notify you when data is deleted; the record simply doesn't exist any more.
Is there a pattern or best practice I can look at? I could brute-force it, i.e. check to make sure each piece of data still exists on the server at some specified interval, but that seems pretty intensive.
I have tried looking solutions online, but was unable to find anything useful.