2

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements.

I have 1-server and n-clients. The server has several tasks to solve with n-subtasks. A task is called A, B, ... and each has 5 (subtask). Each subtask is send to a Queue and gets processed by a client. The results are send back to the server.

  • A subtask is never solved by more than 1 client
  • Subtasks are independent to each other
  • I need to be able to remove all subtasks of 1 specific task from the queue

Any ideas which approach I could use here?

                                                    +-----------+
                                                    |           |
                                                    |  Client A |
                    +----+----+----+----+----+      |           |
                    |    |    |    |    |    +----->+           |
    +-----------+   |Sub |Sub |Sub |Sub |Sub |      +-----------+
    |           |   |Task|Task|Task|Task|Task|
    | Server    +-->+5-B |4-A |3-B |2-A |1-A |
    |           |   |    |    |    |    |    |      +------------+
    +-----------+   |    |    |    |    |    +----->+            |
                    +----+----+----+----+----+      |  Client B  |
                                                    |            |
                                                    |            |
                                                    +------------+
Improve this question
4
  • If you are just looking to setup an async task queue you could use something like celery, no? I assume you've looked at Celery, so if you already have, was there something that didn't fit your needs?
    – Jessie
    Commented Sep 18, 2019 at 20:25
  • AFAIK I can't remove "queued" tasks anymore once they are in the system. That's why I looked into Redis but their publisher/subscriber system is a broadcast which doesn't help either bc I can't see how I can implement a 1:1 relationship of subtask:client
    – Daniel Stephens
    Commented Sep 18, 2019 at 20:27
  • This might be of use: stackoverflow.com/questions/8920643/…
    – Jessie
    Commented Sep 18, 2019 at 20:33
  • It sounds like you would benefit from an actor model, preferably one with built-in support for remote actors. I'm not a Python developer, so I don't know what's available in this space, but something like the Akka framework would be my recommendation.
    – Aaron M. Eshbach
    Commented Sep 18, 2019 at 20:40

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.