I wanted to ask what are the possibilities of maintaining reliability when exchanging messages between microservices, when one of those message is rejected. As of today, we don't want to lose messages, so we are re-queueing these rejected messages. Consequently, there is a risk that the messages will be processed in a different order than expected. As a simple example, let's take a simple architecture of two microservices A and B. A publishes two messages one after the other:

{"productStatus": "PUSBLISHED", "productId": 1}
{"productStatus": "SOLD", "productId": 1}

Microservice B rejects the first message but accepts the second. Due to the fact that we are re-queuing all rejected messages, we are processing the message with the PUBLISHED producer status again, and thus microservice B has incorrect information about the product 1 status. A simple solution that comes to mind is to add objectVersionNumber to each message so that the consumer can tell that he has processed the latest version of the object:

{"productStatus": "PUSBLISHED", "productId": 1, "objectVersionNumber": 1}
{"productStatus": "SOLD", "productId": 1, "objectVersionNumber": 2}

Are there any other approaches to solving this problem? Perhaps the semantics of the message should completely change?

  • For microservice communication, Apache Kafka is a better choice, I recommend you to look into that. Kafka also supports transactions. RabbitMQ is not a bad technology, but it's not the best fit for communication between services. Naturally, the semantics of RabbitMQ and Kafka are completely different, you'll need to study that yourself. While RabbitMQ is a message broker, Kafka is a message log. – Andy Oct 16 at 13:18
  • 6
    There are only two hard problems in distributed computing: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery (I'm sorry, I had to). – Greg Burghardt Oct 16 at 15:46
  • Consider sending a generic change notification without including state in the message body, then asking the recipient to pull the current state from a persistent store/repository (perhaps the message could include a URI so that it knows where to retrieve the state from). This eliminates concerns about timing, ordering or validity of the content of a message since both sender and recipient share a single source of truth. – Ben Cottrell 2 days ago
  • @BenCottrell, bad idea. This introduces the possible necessity to develop additional synchronisation (internal) APIs, which is something you don't want to do when working with microservices (it's more unnecessary work). – Andy 2 days ago
  • @Andy A notification can be sent after the persistent store is updated, so the recipient would be guaranteed to to pick up the current state when it queries the store, no need for any complicated synchronisation – Ben Cottrell 2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.