I'm currently building a service (think of it like Reddit) which allows users to post and comment. It's possible to search for posts within a given radius of your location (e.g. 10km). When comments are posted, the original post's Updated field will be updated to move it back to the top of the "list". Searching isn't ordered by distance, rather it's ordered by Updated.
I'm using a NoSQL database (Mongo) for my data storage, with a view to adding ElasticSearch later for improved search capabilities.
I'm intending to wrap any paging solution into continuation tokens as this will be used with infinite scroll, so random access isn't required, and it will make it easier to swap out the paging implementation in future. The solutions I discuss below would be wrapped as a token.
- Skip/Limit - If a new post is added, then you'll get the same result back in the next page. If it's a popular city, or you leave the app open and a lot of time passes, you'll potentially be very far back in your results.
- Skip/Limit + Date hybrid - An attempted solution to the above could be to also save the initial page requested time, and then ignore any posts that have been created/updated since that date. This helps, but introduces a new problem: if a posts you have already loaded are updated, they excluded by the query (based on date) and you therefore miss posts when you load the next page.
- Limit the number of results you can return (e.g. 1000) and compile a separate result set that contains the thread ids in order. This would work but has obvious issues: compiling the list might take a while, cleanup is required once the result set is no longer required.
- Search by the updated date of the last item in the previous page (i.e. "find all entries older than this date"). With millisecond precision, it's unlikely two posts have the exact same timestamp, but it's theoretically possible to lose results if the last post in a page has the exact same updated date as the first post in the next page (i.e. the first post would be skipped in this scenario).
Many posts and articles I find online use the naive skip/limit solution, but clearly this has issues. I imagine there will be a trade off somewhere with any solution, of course.
My current solution choice is the last one (search by date), but I'm curious if there are other solutions I haven't thought of. What are you experiences of implementing paging? How did you do it?