System for scheduling parallel tasks/callbacks

Question

I am looking for a system for building state machines with timed/scheduled transitions, and with events that happen periodically during a given state. I can implement it myself, though I feel like this is a common enough requirement that there would be a category of software for it.

I don't need a library recommendation, vendor, etc. I just don't know what to call this thing, and thus what to google, because it isn't a "cron" and it isn't a "message queue". I don't know if such a server-side layer even exists currently, but I'm hoping to find an indication one way or the other. If this type of question is a reason for down votes and close votes, definitely indicate that, and maybe point me to another SE, or suggest improvements.

---

The application I have in mind is a timed video game "event" system, like those you might see in an MMORPG (periodic monster spawns, timed escalating waves, failure timer, scheduled event starts), or in a trivia bot (per-question timer, time between questions, round timer), etc. It won't be used for simulations, physics, etc, so I won't be throwing real-time (sub-second) granularity events at it.

The system I have in mind would be some sort of hybrid pub/sub and scheduling system, operating entirely server-side. It would be somewhat like a Javascript setTimeout, except with a more reliable mechanism, probably with a 1 second resolution, and with the ability to pass parameters explicitly when you schedule the timeout. It probably would also have some sort of global monitoring for ops purposes, and maybe would have some sort of support for scaling to multiple systems.

I don't know if I will need to roll my own, or if there are more generic systems that already do what I want them to do. I would prefer to use an existing system if possible. There is some common server tech that is similar, but I don't think it quite does the job.

The system I want operates sort of like Cron, with these exceptions:

1. It should be function callback/micro-process oriented rather than shell oriented.
2. Tasks should probably be executed in a delegate sub-process pool, or in-process of the daemon, rather than spawning a new process per task.
3. It should have a resolution of 1 second (or smaller)

The system I want also operates sort of like a message or task queue, with these exceptions:

1. It should allow for periodic tasks (already possible in some queues, I think)
2. It should support parallel execution for tasks that are scheduled for a given tick, rather than sequentially running them off a queue
3. It shouldn't expect tasks to be long-running, and shouldn't build its interface for long-running tasks
4. It should be just as easy to make recurring tasks as one-off tasks, without having to build task-chaining glue
5. I should be able to cancel scheduled tasks easily and quickly
6. Robustness of preserving and guaranteeing task execution is not a priority over per-task overhead. Vertical and horizontal scalability is way more important than carefully preserving data (at this layer)
7. Tasks should probably be able to share state at some level (so that IPC and task execution latency/overhead is minimized)
8. If the system supports horizontal scaling, I should be able to give grouped tasks an affinity so that they are executed off the same shared state, rather than sent to an arbitrary worker in a pool

My preferred programming platform is python, but I would be fine with a stand-alone system in any language that allows for low-latency server side IPC over a standard protocol, so I can use my server-side language of choice. I'd also be fine with a system that specifically integrates with popular server-side languages, as my task implementations are likely to be a pretty small portion of my larger system.

Having a built-in ability to optionally send events to clients in a push style would also be cool, but definitely does not have to be part of this library or layer.

Answer 1

To be honest, this make me thinks to the JavaScript loop with setInterval and setTimeout and closures though i'm really not in JavaScript.

However here are some problems : if you're tasks need a shared context and can be executed in parallel you may go for some trouble. Events queue aren't mono threaded usually for nothing.

There may be a name for this design, however considering your very specific requirements (functionalities, performance), i don't think that ONE design pattern alone will fit.

The only thing that i see that look like to this is a Workflow Engine. I used Activity, it's a light workflow engine made to be able to process thousands of tasks easily. Tasks have a context which is in the Process. However i don't think this will fit for going below the second.

You were talking about MMO. Do you know you have two different "clocks" : one is the famous Frame Per Seconds that everyone knows, the other is the clock that handle communication with the game, each of them are independent. They basically schedule their own tasks separately because even if they're basically Task in the most Abstract terms, the requirements for them are totally different.

Having in the same queue tasks that must be executed 10 times in a second and other only once per hour with all the other requirements would make your "queue" (or whatever) a god object in my eyes. Really too complex and specific to put a name on it.

So you may need to split your problem and change the design of your tasks accordingly in multiple independent problems :

• Tasks with shared context
• Tasks with multiple execution within the second.
• Independent Tasks with a long interval between execution.

Each of them could use the same interface, say a facade (or a proxy ?) to subscribe in fact to different queue according to the parameters given. Each of the queue would fill only the requirement it was made for

• one for high performance,
• one for shared context,
• parallelism.
• some mix : parallelism + shared read only-context ?

In the end, i make each components having the less responsibility possible to be able to focus on what is more important for each of them.

Answer 2

I've recently been designing something similar. What did you end up using/doing?

In my case I'm calling it Event Scheduler although it fundamentally is the same as what you're describing, which could also be called a Task Scheduler (exactly what Celery labels itself).

The only issue that prevents me from using Celery directly is that I need to schedule tasks far into the future and I need to be able to cancel and update them before the trigger time.

My current solution involves a Cron polling a table of scheduled events and triggering the proper tasks/functions although there are open questions regarding the design of these tables to optimize inserts/lookups.

High level description:

EventScheduler module

Events Table      -> sent, archived events just for record-keeping reasons

EventsQueue Table -> unsent, indexed by `trigger_time`, polled by cron every minute

# using a PostgreSql DB