Imagine there is a stream of requests for about 500 types of work. There can be say 5 workers in parallel. One type of work should be executed by at most one worker at the same time. The requests for a specific type of work should thus be done sequentially. So further requests for that active type of work should be enqueued till that current work is finished.
So, when the work of 1 type is being executed, only further requests for that type of work are enqueued. In the mean time all other types of work can continue.
In our Spring environment I tried a number of solutions for solving this situation like:
- 2- Each type of work is assigned a queue. So there will be many queues. Sometimes a type of work will disappear, so then the specific queue is deleted. So any type of work is out on the specific queue.
- 3- Have 1 message queue dispatching to a limited set of worker queues. That requires quite some logic for dispatching. There should be a standard solution for this problem.
- 4- Have 1 message queue in combination with database tables that contain the 'suspended' messages. The problem of this solution is that it is hard to process the messages of one type of work in the right, sequential order.
- 5- A bit awkward: just use a database table with workers reading the first (oldest) matching message. It works, but not ready with setting the right transaction boundaries. Is quite expensive in relation to database access
We started with experimenting with the dynamic creation of queues. This feels like very dynamically creating/suspending/resuming infrastructure. Is this right way to go?
The current solutions don't proof to be a good, solid solution.
How to model this in a good way? Is the option of dynamic creation, suspending and resuming of queues a good idea?