I'm creating a framework that needs to execute certain tasks, I'll use building a car as an example for this. So, an individual task could be something like
weld metal or
screw bolt. But there are also collections of tasks, I call them jobs. Each job can have X amount of tasks, so combining these two tasks could get us a job
build an engine:
build an engine +-- weld metal +-- screw bolts
Additionally, jobs can have sub-jobs:
build a car +-- build an engine | +-- weld metal | +-- screw bolts +-- build a frame | +-- weld metal | +-- screw bolts | +-- paint +-- combine engine and frame
So basically each node of my tree is a job, and each leaf is a task. This is easy enough, here's what my classes would look like:
class Task: def do(self): raise NotImplementedError('abstract method') class Job(Task): def __init__(self): self.tasks =  def do(self): for task in self.tasks: task.do()
And now I'd just subclass
The problem arrives when we want to parallellise this. I need to somehow tell each task (and job) whether it should block the execution, or run in a separate thread and not block it. Sometimes my jobs also need results from other jobs, while those jobs may run in parallel to each other. For example in the above car building:
weld metalmust block, since you can't screw before the metal is welded.
build an engineand
build a framecan be ran in their own threads (and within those threads the
screw boltsmust block the particular thread).
combine engine and framemust wait for
build an engineand
build a frameto finish.
Honestly I'm not too sure where to start, initially I thought of using
Job.do(), but I'm not sure how to block only some jobs and not others, and another issue is that sometimes a task must be ran "alone" without even being inside of any job, i.e.
WeldMetal(block=False).do() must be valid.
I ended up using this at first:
class Task: def do(self): raise NotImplementedError('abstract method') def run(self): if self.block: self.do() else: threading.Thread(target=self.do).start()
And this works for most cases, but the issue comes when
combine engine and frame needs to wait for both
build an engine and
build a frame to finish.