I have this bot that I've created for Twitter. It's written in Python. It lives on my Raspberry Pi and I've set up a cronjob to run it every 60 seconds.

Part 2 of the project is to build a web app that I can use to configure the bot - instead of messing with the code directly.

In Part 3, I want to extend the web app with Twitter's 3-legged auth to allow other people to use the service with their Twitter accounts and configure it to their liking.

The bot takes a few seconds to complete its execution. So if I decide to use one script with one mega loop that linearly runs through every user, it will take awhile to finish. It wouldn't be a problem if time wasn't an issue, but it is. The bot scours sources and forwards to Twitter anything that's new since it ran last (60 seconds ago).

It seems like with those constraints that I need to be able to run this service in parallel for each user or some x number of users, but I don't know of a fast, efficient way to do that.

  • I don't think it makes much sense to create a new script with their credentials and configuration setting, and update them when requested, for every user and set up a cronjob for each user's script.
  • I don't think I can use one script to linearly run through every user, because each execution takes a few seconds to complete, so total time would grow with each new user.

I wonder if there are services, frameworks, APIs, or some other architecture styles that would work to scale this up. What should I do?

  • 2
    First, identify where the time is being spent. If the vast majority of time is spent scouring sources, and it is the sources that are slow, changing the code may not do much. – Robert Harvey Aug 18 '16 at 15:18

Move the configuration parameters to a back end database, and create the web UI to manipulate those settings which the script reads in each iteration (in case the parameters have been changed)

For part 3, start looking at the twitter API and the various python wrappers that are out there.

To improve the frequency that the script executes for each account, look into multi-threading and thread pooling with python and put the work of polling and updating each account into a separate thread. The work is mostly IO bound, so don't hesitate to create more threads than there are processor cores.

Look into the various service wrappers that might be out there which you can leverage. I am sure there are some. A service wrappers job is to wrap around your code with some monitoring capabilities, and at a minimum detect when the code has crashed or become unresponsive and restart it.

You might consider capturing tin the back end database each time each account is updated, so that your users have a way of knowing when their data was last polled. Wouldn't be a bad idea to give the ability to trigger an update from the web UI page.

| improve this answer | |

First ditch the cron job. Convert the program so it has a long running loop. You don't need 60 emails an hour to tell you it's broken.

To scale, compute a stable hash of the username (or any other stable attribute) of the user. Decide how many copies of the bot to run (you can change your mind about how many). We'll call this number N.

Take the hash value modulo N - this tells you which instance ("shard") of the bot should deal with that user.

| improve this answer | |

why does it need run in Raspberry Pi instead of proper server? I think parallel processing capabilities are quite limited. Anyway, here are my idea, I don't know if these works in Raspberry Pi.

First step, parameterize the python script so a command-line argument specify which user should be run. Second, make the web app manipulates cron entries, so each new user means a new cron entry, with the userid specified in the command line.

| improve this answer | |

You can look at AWS Lambda service combined with AWS API Gateway. It performs load-balanced execution of code based on various events like database change or incoming HTTP request.

| improve this answer | |

Firstly, try run it on a VPS or something and see how that works out. The Raspberry Pi might be great, but it's not powerful.

Secondly, change the Python script to use parameters to take authentication and then update the cron job's to use the new parameters.

Thirdly, either add multiple cron job's and have them all run as their own python instances, it's horrible but most of the latency is probably from whatever service you're polling for info.

The better way to do this would be to work out how many users one Python script could handle per minute, and use a number slightly lower than that to be the limit per Python instance, and creates instances of the Python script (from another, controller script) depending on how many users there are that need to be checked. This way, if the number is 15 users per minute, and you have 18 users, it will create another Python "daemon" or handler to handle the other users. You have to ensure that the "daemon" style thing can communicate with all the others, and that the controller assigns what users each daemon is responsible for/etc.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.