In the following I describe a very very simple scenario to keep things simple. I am looking fortry to develop a system how to properly combine the machine power of severalwhere many machines to support a servercan help another one (server). I initially started using RabbitMQ, then tried reddisRabbitMQ and Redis, but in both cases they don't support a feature the other hasfulfil all requirements.
In my scenario there isI have 1-server and n-clients. The server has several several tasks to solve which is made out ofwith n-subtasks. Let's imagine theA task is called calculate-1A and has 1000 formulas to solve5 (subtask). Each formulasubtask is send to a Queue system where each client can pick a formula, calculate it and sendgets processed by a result backclient. Never ever do two clients calculateThe results are send back to the same formulaserver.
- A subtask is never solved by more than 1 client
- The tasks are independent to each other
- Subtasks are independent to each other
- The order is not important
- I need to be able to remove all subtasks of 1 specific task from the queue
Any ideas which approach I could use here?
+-----------+
| |
| Client A |
+----+----+----+----+----+ | |
| | | | | +-------->+ |
+-----------+ |Sub |Sub |Sub |Sub |Sub | +-----------+
| | |Task|Task|Task|Task|Task|
| Server +-->+5 -B |4 -A |3 -B |2-A |1-A |
|1 |
| | | | | | | | +------------+
+-----------+ | | | | | +-------->+ |
+----+----+----+----+----+ | Client B |
| |
| |
+------------+
My approach with RabbitMQ: RabbitMQs queue system does almost what I want. But cancelling a task needs to be possible, and I can't remove individual tasks from a queue once I pushed them there.
My approach with Redis: The publisher/subscriber approach is cool but only broadcast messages are supported. I can't limit a single task to a single machine.
Any ideas which approach I could use here?
