I'm setting up a service that is meant to query, via a rate-limited API, a number of posts in real-time. Now, some of these posts can be several months old, and they should really only be queried, say, a few times a day once they reach an age where they're unlikely to see new activity: most of my queries should be spent polling new posts.

How can I structure an algorithm to pick the next post to query that weights newer / more active posts higher than older ones, while still querying older posts at a reasonable sample rate?

  • Doesn't this depend entirely on how the users of this particular service tend to update stuff? My first thought would be to gather some data on update histories of various posts for this particular site, make a scatter plot of age and time-to-next-update and do a regression analysis. – Ixrec Apr 25 '15 at 15:50
  • For each post have a list of "things" (people/members?) to notify if/when the post changes; and for each "thing" have a list of posts that have changed. Whenever a post changes, add the notifications to each "thing's" list. Whenever you need to send anything to a "thing", append its list of posts that changed to whatever you're already sending (and set the list to empty). No polling, no delays, no searching through thousands of posts to see what changed, no/minimal extra packets across the network. – Brendan Apr 25 '15 at 16:12
  • @Brendan "Tell the API provider to design an infrastructure for web hooks" isn't an algorithm. – Stuart P. Bentley Apr 25 '15 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.