We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast message.
We are evaluating several options of how the producers are going to publish to the queue.
Evaluated solution proposals
I've come up with three options. Here are the pros/cons of each I can think of:
1. Producers directly connect to the SQS endpoint and publish the message.
Pros
- The responsibility of availability, latency is shifted to SQS endpoint.
- SQS provides IAM authorization.
- No need to maintain any infrastructure.
Cons
We cannot control the message content that goes to the queue. Let's say there is a bug in a producer that leads to invalid message being put in the queue which has to be handled by the consumer. Some options include isolating bad messages in a dead letter queue or ignoring it since it is not a valid message altogether.
If we wanted to switch the distributed queue provider, then we would have to change all the producers to publish to a different endpoint.
2. Create a REST service with an API that validates the request and forwards it to the queue. All producers call the API.
Pros
- Can validate the message at the API layer before putting it in the queue.
- Can switch to a different distributed queue or processing mechanism without having to change the producers.
Cons
- Cost associated with creating and maintaining a service that does something trivial as validation and putting the message to the queue eg. infrastructure cost.
- Another layer of network indirection just for validation and cleaner data contract.
- Maintaining availability. Adds one more point of failure. Although this service does not have any dependencies besides SQS, we do have to take responsibility of the availability.
- In option 1, this concern would be handled by SQS.
- Have to implement authz/authn on the API.
3. Expose a client library to the producers that connects to the queue endpoint and publishes the message.
Pros
- Gets the best of both option 1 and option 2. We can add validation logic in the client library and expose the appropriate interface in the code. We can also switch to a different distributed queue or processing mechanism by making code changes to the client library and getting the producers to use the new version.
Cons
- The library is going to be programming language specific. If we have producers in different languages, we may have to build language specific client.
I am leaning towards option 3. It has the pros of options 1 and 2. Also, most of our producers are microservices written in a particular language and I don't think we will be experimenting with newer languages anytime soon.
Questions
- Am I missing some options or pros/cons?
- Are there any best practices for multiple producer to single consumer communication?
- Are there cases where Option 1 or Option 2 would be more appropriate?
