I have this bot that I've created for Twitter. It's written in Python. It lives on my Raspberry Pi and I've set up a cronjob to run it every 60 seconds.
Part 2 of the project is to build a web app that I can use to configure the bot - instead of messing with the code directly.
In Part 3, I want to extend the web app with Twitter's 3-legged auth to allow other people to use the service with their Twitter accounts and configure it to their liking.
The bot takes a few seconds to complete its execution. So if I decide to use one script with one mega loop that linearly runs through every user, it will take awhile to finish. It wouldn't be a problem if time wasn't an issue, but it is. The bot scours sources and forwards to Twitter anything that's new since it ran last (60 seconds ago).
It seems like with those constraints that I need to be able to run this service in parallel for each user or some x number of users, but I don't know of a fast, efficient way to do that.
- I don't think it makes much sense to create a new script with their credentials and configuration setting, and update them when requested, for every user and set up a cronjob for each user's script.
- I don't think I can use one script to linearly run through every user, because each execution takes a few seconds to complete, so total time would grow with each new user.
I wonder if there are services, frameworks, APIs, or some other architecture styles that would work to scale this up. What should I do?