2

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast message.

We are evaluating several options of how the producers are going to publish to the queue.

Evaluated solution proposals

I've come up with three options. Here are the pros/cons of each I can think of:

1. Producers directly connect to the SQS endpoint and publish the message.

Pros

  • The responsibility of availability, latency is shifted to SQS endpoint.
  • SQS provides IAM authorization.
  • No need to maintain any infrastructure.

Cons

  • We cannot control the message content that goes to the queue. Let's say there is a bug in a producer that leads to invalid message being put in the queue which has to be handled by the consumer. Some options include isolating bad messages in a dead letter queue or ignoring it since it is not a valid message altogether.

  • If we wanted to switch the distributed queue provider, then we would have to change all the producers to publish to a different endpoint.

2. Create a REST service with an API that validates the request and forwards it to the queue. All producers call the API.

Pros

  • Can validate the message at the API layer before putting it in the queue.
  • Can switch to a different distributed queue or processing mechanism without having to change the producers.

Cons

  • Cost associated with creating and maintaining a service that does something trivial as validation and putting the message to the queue eg. infrastructure cost.
  • Another layer of network indirection just for validation and cleaner data contract.
  • Maintaining availability. Adds one more point of failure. Although this service does not have any dependencies besides SQS, we do have to take responsibility of the availability.
    • In option 1, this concern would be handled by SQS.
  • Have to implement authz/authn on the API.

3. Expose a client library to the producers that connects to the queue endpoint and publishes the message.

Pros

  • Gets the best of both option 1 and option 2. We can add validation logic in the client library and expose the appropriate interface in the code. We can also switch to a different distributed queue or processing mechanism by making code changes to the client library and getting the producers to use the new version.

Cons

  • The library is going to be programming language specific. If we have producers in different languages, we may have to build language specific client.

I am leaning towards option 3. It has the pros of options 1 and 2. Also, most of our producers are microservices written in a particular language and I don't think we will be experimenting with newer languages anytime soon.

Questions

  • Am I missing some options or pros/cons?
  • Are there any best practices for multiple producer to single consumer communication?
  • Are there cases where Option 1 or Option 2 would be more appropriate?
4
  • Does message ordering matter for you? Jul 29, 2020 at 13:04
  • No, it does not
    – kashive
    Jul 30, 2020 at 19:50
  • By option 3, are you thinking pub/sub? Aug 29, 2020 at 6:09
  • 1
    Why do you need the queue? can't the consumer just be an HTTP server? If you don't care about response, you could check UDP, but that doesn't need queues. Aug 26, 2022 at 7:11

2 Answers 2

2

Concerns

According to my understanding these are your main concerns:

  • Producers are publishing data without thinking, that's why data sanitization is needed
  • The operational and maintenance cost
  • The underlying queuing mechanism might change in the future

Where to put data sanitization?

As you have listed you can put this logic into

  • Producer side (Option #3)
  • Consumer side (Option #1)
  • Into an intermediate layer (Option #2)

This concern could be evaluated by asking the following questions?

  • What is the percentage of good / garbage message?
    • For example: If there is lot of trash data then filtering should be place as near to the producers as possible
  • What amount of data are we talking about?
    • For example: If the producers are pushing way more data than the consumer can process (fast producer - slow consumer problem) then you might to think to introduce throttling / sampling
  • How dynamic is your filtering logic?
    • For example: If it uses a bunch of rules that are parameterized with database records then a separate tier would make the most sense.
  • etc.

Vendor-locking vs Being technology agnostic

  • If you want to minimize maintenance cost then you would choose a PaaS solution, which would make it really hard to change that decision later
  • If you choose to adapt the anti-corruption layer and introduce a thin proxy then changing the underlying queuing might be an easier task
  • If you try to follow the smart endpoints and dumb pipes guidance then you would want to minimize the usage of PaaS specific functionalities
  • etc.
0

This is quite an unusual setup - the general concept of publisher/subscriber is that one publishes to many subscribers, not the other way around.

When you're publishing these messages from different systems, are they "events" (something that's happened and you're informing other systems about) or "commands" (something you would like to happen on the consumer)?

What is the "source of truth" in your system? For example, your Customer API should be the sole place where customers are created, updated and deleted. That API accepts commands to do this work, validates and processes them. It can then emit an event that this has happened to other systems to subscribe to.

If you have multiple publishers all sending events of the same type - what if they disagree? If you're publishing commands - you need to seriously consider whether this pattern is correct - you can't "fire and forget" a command, it needs to be validated and processed, and the result returned to the requester.

If your consumer is a genuine consumer (i.e. it just wants to know what's happening, it's not there to decide whether what's happening is right or wrong) then proceed with the next part. If not, you need to go back to the drawing board in terms of architecture. What are you trying to achieve by using a message queue? And is a decoupled, async integration really the right choice?

As for your three options - these are your trade offs. Whatever you do, have a standard contract for publishing to the queue/topic. Manage this contract carefully to keep everything lined up.

  1. How likely are you to change message queue provider? It should be pretty easy to properly separate your code concerns to make it relatively easy to change in future (albeit across multiple services).

  2. If you write an API, this now contains rules about your validation business logic, and must be kept up to date, along with everything else. You'll also need to write clients in each service to talk to it, and keep them in line.

  3. The client library sounds like a nice idea, but the reality is likely to be less pleasant. Language lock-in 100%. It'll be a little easier to change your various publishers to fit it, but you still need to keep them all up to date if the contract changes, and the library needs to be flexible enough to handle all the different ways those systems might act or structure their data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.