Scheduling of parallel I/O-bound tasks (Backup solution)

Question

I want to implement a backup solution in Python where a Backup-Server initiates backups on a number of virtual and physical servers (one server = one backup task). Disregarding the details of the actual backups tasks I am concerned with the scheduling/multiprocessing part for now.

The constraints I have are:

1. Only backup two servers at once (e.g. have at maximum two backup threads running at once)
2. Don't backup two servers on the same physical machine (oftentimes multiple virtual servers share a common hardware machine) at once.

Since I am not too experienced in multiprocessing in Python I am wondering what an optimal Python solution would be. The following came to my mind:

• Have a thread for each backup-job (e.g. for each server) and use a threading.BoundedSemaphore to ensure only two are running at once. Use more semaphores/conditions to ensure that multiple threads are not backupping two servers on the same physical machine.
• Have exactly two threads that are running all the time and retrieve their tasks from a queue. Simultaneously the queue would have to make sure no tasks on the same physical machine are handed out at once (e.g. skipping/reordering tasks at times) . I would probably do this by subclassing Queue.PriorityQueue to add the additional constraints.

I am leaning towards the second option but I am not sure whether or not a queue is the right data structure to hand out the tasks to multiple working threads. I don't need to add tasks to the queue at runtime (which a queue allows) and I need a bit of logic to hand out the tasks rather than just process them in a linear order. Is there is a better (standard) data structure for this?

I would be thankful to hear some thoughts from more experienced programmers.

Answer 1

I'd have a queue per physical machine.

Thus a backup process would take at most N=2 tasks, and make sure it does not pick a task from a queue for which another task as already running.

(N could be easily adjusted when needed.)

Answer 2

To make your life much more easier, and with much less boilerplate code, i suggest you take a look at Advanced Python Scheduler

Its very easy to start working with, and it will save you all the time and trouble of managing queues, threads, and scheduling.

Can run in multiprocess or in threading modes. you can limit the number of tasks executed at once by easy configuration. see examples here

It gets even better, it wont execute the same task again if already previous task is still running, it will wait until finished by setting max_instances: 1.
if you need state, you can make use of jobstore to make your tasks persistent.

code example:

from pytz import utc

from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.jobstores.mongodb import MongoDBJobStore
from apscheduler.jobstores.redis import RedisJobStore
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from apscheduler.executors.pool import ThreadPoolExecutor, ProcessPoolExecutor


jobstores = {
    'mongo': MongoDBJobStore(),
    'redis' : RedisJobStore(),
    'default': SQLAlchemyJobStore(url='sqlite:///jobs.sqlite')
}
executors = {
    'default': ThreadPoolExecutor(20),
    'processpool': ProcessPoolExecutor(5)
}
job_defaults = {
    'coalesce': False,
    'max_instances': 3
}
scheduler = BackgroundScheduler(jobstores=jobstores, executors=executors, job_defaults=job_defaults, timezone=utc)  

def backup_job(**kwargs):
    print('doing backup stuff')


scheduler.add_job(backup_job, 'interval', seconds=600, jobstore='redis'
                  id=1, name='backup', kwargs={'key':'value'})

hope it helps