I'm making an Rails app that in theory should scrub new posts from the users facebook and twitter accounts and put them in a timeline for an analyst to analyze and determine if they are good or bad posts, if the user posted "bad content" the analyst marks it as so and warns the user.
I've made the authentication with omniauth and have made both facebook and twitter apps, users can already sign in and in return I get a oauth key and secret, with these I can make the respective API calls.
Now for the real question, how am I supposed to check for new updates? I've thought using resque or sidekiq to queue each user's facebook and twitter accounts periodically, but wouldn't it be too much if I had many users, making a new API client (I'm thinking Koala and Twitter gems) with each user's key and secret every certain time?
What's the best way to do this?
And once I got the users' posts should I store them in the DB or just put them in another resque or sidekiq queue for them to be analyzed and discarded once they are?