Architecture for worker pool with different task complexities and sizes of boxes

Question

I'm sorry if I'm using any wrong terminology here.

I'm trying to design an architecture where there can be big and small tasks (e.g. processing big or small images). Big tasks can only be handled by big boxes while small tasks can be handled by either big or small boxes. The big tasks can finish while there are still small tasks to run. It doesn't make sense to immediately scale down the big boxes because they take a while to boot, and we always want at least some number running. However, it would be nice if the big boxes could start processing small tasks while they wait because they cost a-lot.

1. I was thinking one way could be to have a completely separate service for each then if the big tasks were done it could start consuming from the small queue. (see image below) However, this seemed troublesome with 'at least once' delivery and potentially other issues

2. I thought a separate way could be to have both in the same service, and talking to the same database, but utilize the 'inbox' messaging pattern so the big box would read the other 'inbox' if there were no messages in it's default. The thing that seemed strange to me about this was that different instances in the same service would be configured differently. That's not something I've done before so it smells fishy to me.

3. A third option I thought of would be to have two separate services, but both talk to the same database. Again, I've not done that before so it just smells fishy to me.

Is there a 'typical' way of handling this problem? Should I just be avoiding this all together and just forget trying to have 'big task service' process small tasks?

Answer 1

Superficially, it seems to me that what you architecturally describe as "big instances" or "small instances" might in fact be "task monitors." (My term.) These are processes whose job it is to curry through the single(!) task-queue, to opportunistically select which unit-of-work they will now claim.

The work is divided into "classes," but this distinction is only expressed in an element of the database entry in a single common database table. All of the incoming units of work are placed in the same table, each one bearing a class-ID.

Each task-monitor is provided with a prioritized class selection list. For instance, a list of "ABC" means that it will select a class-A unit of work first if there is one, then "B" and so on. In real time, in an atomic database transaction, it cherry-picks through the list and then claims the first one. A "small" server might not have the letter "A" on its list at all.

Having claimed a task-queue entry, the task-monitor process then spawns a child process to carry out the work, and waits for it to complete. It then updates the task-queue entry with completion status and repeats the process.

Task-monitor processes running on different computers would have different class-selection lists according to their own hardware capabilities or intended purpose. The task-monitor process itself is generic and used everywhere. The child processes which they spawn vary considerably.

Therefore, the "task services" (computers ...), in your diagram, are each hosting a certain pool of "task monitor" processes which are from time to time each poring through the list of available work, choosing which one to pursue next. Simple database-transaction protocols provide the serialization. The number of task-monitors that are hosted on each computer reflect their maximum capabilities: how many units of work they can be expected to support at one time.

Answer 2

Based on @Mike_Robinson's comment I was able to better research the problem.

Essentially I've described a High Performance Compute (HPC) cluster with different instance types. This is referred to as a heterogeneous compute cluster For historical on prim clusters it seems a common tool for workload management is slurm. AWS has AWS ParallelCluster which seems to have a slurm base and has support for heterogeneous clusters.

This is also a similar problem solved by AWS Batch

Searching for task scheduler system design you can find a number of resources like drop box's task framework. There are a-lot of references to this as an interview question as well as a number of discussions on youtube with these keywords