I have an application made up of multiple processes/workers/services which need to send messages to each other that represent units of "enqueued tasks" to be done. I am trying to find the best pattern to use for the type of communication these workers need to perform.
These are the requirements:
- Communication is asynchronous. One worker enqueues a task for another worker to do, and doesn't block for its completion or require a synchronous response.
- The workers do not share memory.
- The messages need to be persistent and durable. They should be stored when the application is shut down and resumed when the application starts back up.
- Messages should be "unique" in the queue. That is, if one worker needs another worker to perform a task, but that task is already present in the other worker's queue, then it shouldn't be enqueued again. This is to avoid generating a huge backlog of redundant tasks in the queues of slower workers which were generated by faster workers.
Points 1-3 just scream AMQP. But then point 4 is the problem, because I don't know of an AMQP implementation with a straightforward solution for enforcing uniqueness of messages (I know RabbitMQ doesn't).
I'm wondering if the fact that these messages queues actually represent enqueued tasks to perform, and need to be unique, means that there is a more suitable pattern than MQs.