I have the following requirement:
There are about a 100 recurring jobs that needs to run every one minute.
This job has an IO intensive phase, followed by a CPU intensive phase, followed again by an IO intensive phase (total amount of time taken to run the job can vary from 2 seconds up to 50 seconds).
Its extremely critical that all the jobs runs and that no two instances of the same job can run at the same time (so if a job takes longer than 1 minute, then the next job should not start). However, two different jobs can run at the same time.
Its also extremely critical that there is no downtime, hence the jobs should not rely on one machine being up.
Finally, as much as possible, we need to be able to use in memory caching (especially during IO phase).
Right now here is the architecture:
There N boxes where the jobs have been split. Each job has one dedicated Master and one dedicated Slave.
The Master and Slave use an in memory Quartz scheduler to run every 15 seconds. The master checks if the last time the job ran was over a minute ago, and if so tries to grab a lock in Redis. The Slave also does the same, except it waits for 1 min 30 seconds (there by ensuring master has the priority).
After grabbing the lock the job is executed and then the lock is released and the time the job finished is persisted to check against the next run.
Since most of the time job is run on the dedicated Master, we can utilize in memory caching extremely effectively. Furthermore, the in memory cache size is small since it mostly caches data just for the jobs it is the master of.
However, the issue with this is since the jobs are IO bound, then CPU bound, and then IO bound, running the entire job on one box is very expensive and severely underutilizes the capacity.
Instead we wanted to do the following:
Have each phase run on dedicated set of machines. A master scheduler kicks off each of the 100 jobs.
The boxes dedicated IO intensive starts then sends over all its data dedicated to the nodes for CPU intensive phase which then sends over the data to the third phase and so on.
That way we can effectively use our resources.
What architecture / open source tools would you suggest to run such a workflow?
We need to ensure that the system is highly available and fault tolerant. Ensuring we can have a In Memory cache will be a significant plus. Furthermore, the data generated between stages can be large, so we need to reduce serialization and network costs (which is not present today).
Initially we thought of using Spark, however, our Jobs are extremely business critical. Will Spark provide the guarantees required? Spark seems a better fit for handling massive amount of data generated (which we don't have, a few jobs, but very critical)