Alternatives to cron jobs or other ways of improving scheduled task performance

Question

I'm working on a social networking website where users gain ratings after specific actions taken and proper conditions are met on the tables. The ratings are calculated depending on 'total hours worked' and 'total points gained'. The cron php has an sql that INNER JOINs 7 tables. To keep this up-to-date we run a cron to check the table entries once a day to update values.

This is just one cron of 15 in total. Some of them only sends emails with Mandrill, one sends a newsletter and there will be more newsletter in near future.

Currently, when there is an active event on the site (it's kind of an event networking site but with administration and onsite management features), around 2000+ users and their data are involved in these processes. But there will be more and more soon.

We had two server crashes that really ruined the server because of two crons before, one of them is the one which updates user ratings. The second cron that was having problems was updating the total hours a user worked in two tables. We will need to have even more scheduled tasks for other data like these to be updated automatically.

After some research I've found The Fat Controller which is -

A parallel execution handler, used to repeatedly run other programs, usually scripts, a bit like CRON. It was designed to handle the execution of scripts which perform background processing for websites which generally need to repeat, react to how much work there is to do at any one time or run as a daemon.

(As stated here - http://fat-controller.sourceforge.net/index.html )

Any ideas on how the performance of running 5 php scripts previously used as cron jobs would be compared to running them as cron jobs? Also, some of the cron jobs we currently have need to run in specified hours/days (i.e send mail to a user - not weekend - at 9.30 etc.) is this possible with a tool like the fat controller? Or is it possible to set the task to run once a year?

Any recommendations on the best way to handle this kind of processes would be really appreciated.

Answer 1

I've always liked cron. I think it's easy to understand, and (like most *nix apps) it only does one thing - launch scripts when the current date/time matches a pattern.

What those scripts do, however, is a completely different story. If they are well written and tested, they can be awesome - critical components of your day-to-day operations.

If poorly written, they can (like any app) can bog the system or worse.

Most of the cron replacements that I've seen were written to be easier to use, not to be "more stable" or "more performant". cron itself is very stable and since it doesn't do any heavy lifting, performance really isn't an issue.

If I were you I'd take a good look at the scripts you're asking cron to run and I'd also look at your database. You might have some expensive database calls, be missing an index, etc.

Answer 2

I run a Rails server that has to handle quite a lot of background processing. We use different approaches depending on the job wee need to do and the amount of processing they take or how we need to schedule them.

To start, your jobs should be able to run concurrently. That's basically seeing every job as a user and making sure to use the same technics you would use for multiple users like transactions and record locking.

Basically there are three approaches:

Simple cron jobs. We use this for jobs that run very regular (every hour, once a day) and that need A lot of processing (for some weird reason running a Rails script from the command line and the same script inside aRails controller the latter case sometimes uses much more memory, also runs a bit slower)

The second approach is what you call the 'fat controller'. This controller is called by a cron job (just running a simple curl or wget from cron.hourly). That's for some simple tasks where we run things at a specific time, or maybe only if other jobs do not run. (I use a simple trick here, when the controller starts it will write a temporary file and deletes it when finished, if the file is present at startup it will not run at all, avoiding some conflicts if some data import takes longer than expected).

Third variant is a quequ. Basically a table where I write some requests that can be performed later or at night (sending emails, converting images...). Another cron job will then start to work on this queue at a given time. (There are several nice libraries for Rails that help organizing this kind of work.)