0

I'm setting up a service that is meant to query, via a rate-limited API, a number of posts in real-time. Now, some of these posts can be several months old, and they should really only be queried, say, a few times a day once they reach an age where they're unlikely to see new activity: most of my queries should be spent polling new posts.

How can I structure an algorithm to pick the next post to query that weights newer / more active posts higher than older ones, while still querying older posts at a reasonable sample rate?

Improve this question
3
  • Doesn't this depend entirely on how the users of this particular service tend to update stuff? My first thought would be to gather some data on update histories of various posts for this particular site, make a scatter plot of age and time-to-next-update and do a regression analysis.
    – Ixrec
    Apr 25, 2015 at 15:50
  • For each post have a list of "things" (people/members?) to notify if/when the post changes; and for each "thing" have a list of posts that have changed. Whenever a post changes, add the notifications to each "thing's" list. Whenever you need to send anything to a "thing", append its list of posts that changed to whatever you're already sending (and set the list to empty). No polling, no delays, no searching through thousands of posts to see what changed, no/minimal extra packets across the network.
    – Brendan
    Apr 25, 2015 at 16:12
  • @Brendan "Tell the API provider to design an infrastructure for web hooks" isn't an algorithm.
    – Stuart P. Bentley
    Apr 25, 2015 at 20:54

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.