Is it possible for a queue to have multiple consumers?
I am working on a system where a single queue needs to be accessed by multiple consumers for different micro-services. Is it possible for the queue to retain data until after a consumer has consumed it or are there other possible methods for multiple consumers to consume data from a queue?
A queue should (except for specific cases) be treated as an inbox for each application, so each consumer interested on some message type should have its own queue.
Don't forget that AMQP promotes a producer/consumer pattern, so each app would "steal" a message to other, this is very nice when the consumers are many replicas of the same microservices, while if different microservices are interested in same messages they have to declare that want message copies on thier queues creating a binding. The binding is a rule that implements the following behaviour
given the exchange < E >
when a message is sent on < E > with routing key < k >
then put a copy on the queue < Q >
So the queue is just a buffer, the routing key resembles the content of the message
when the messages are put on the queue, the consumer receives is and executes some business logic. If the logic has been correctly executed, the service doesn't need the message anymore. If an error occours, you have to decide if there is need to retry the execution (hopefully with a different healthy replica) or give up and save the message for further post-mortem troubleshooting. This is accomplished with ack/nack packets (and the dead letter exchange) that the consumer sends to the broker. Most of AMQP libraries do the job automagically: Spring AMQP sends a nack if an exception has been thrown through the message listener associated to the queue
I'm late to the party, but maybe this is still useful for others.
A fanout exchange does what you need.
Create one queue for each application/microservice. Create a fanout exchange. Bind each queue to the fanout. Now each message sent to the fanout will be delivered to each queue, i.e. the same message will be consumed by all microservices.
,--------, ,---------------,
,-->| Queue1 |----->| Microservice1 |
| `--------` `---------------`
,---------, ,==========, | ,--------, ,---------------,
| message |----->|| Fanout ||--+-->| Queue2 |----->| Microservice2 |
`---------` `==========` | `--------` `---------------`
| ,--------, ,---------------,
`-->| QueueN |----->| MicroserviceN |
`--------` `---------------`
can a queue have multiple consumer ?
Yes it's possible that multiple consumers take messages from the same queue. But every message is only consumed once by one consumer
is it possible for the queue to retain data until after a consumer has consumed it
No the messages are not visible for the other consumers if one consumer started consuming it.
I am working on a system where a single queue needs to be accessed by multiple consumers for different micro-services.
I think you need a queue per cause/topic that has to be handled
Yes, but it's not a queue, it's then called the publisher/subscriber pattern.
Queue - every item gets processed once. If there are multiple agents, every work item still gets processed once. Example: Payment processing
Pub/Sub - every item gets processed by every client/agent. Example: Customer change of address.
