We have multiple microservices communicating over MQ. As MQ messages are the interface/contract between the services, whenever we make changes to the MQ message published by a service we need to make the same adjustments on the services which consume the message.

As of now, the number of services is small enough so that we know which services communicate with each other, and can keep the MQ message contract in sync between them. But as the number of services grow this becomes harder.

Option 1: Break things first, then fix it

I've been thinking maybe of implementing some kind of health check. Let's say service A during operations may emit message type X, which is consumed by service B. Service A could then on startup emit a health check type of message, something in the lines of a message X dry-run. When service B receives this, it simply verifies that the message is according to contract. If not, for example if service A have remove a critical field in the message, then service B will reject the message which in turn will end up on a dead-letter exchange, which again will trigger a warning notification to the devops staff.

This approach won't prevent us from deploying non-compatible message types, but will notify us pretty much instantly when we do. For our use case, this might work due to our very small number of developers and projects, so if we break things like this we'd be able to fix it quite quickly.

Option 2: Early probes

A variation over this might be that we start versioning the MQ message format (which we probably should and will do anyway). Then, when service A plans to upgrade from version 1 of message type X to version 2, server A could early on start emitting "dry-run" type of version 2 of message type X. This would cause service B to drop the message. Say this happens a few days or weeks before service A perform the actual switch from version 1 to version 2, then the devops team will have time to add support for version 2 in the mean time.

Option 3: Manually detecting conflicts before deployment

Another approach would be to have some way of detecting - before the actual deployment - that service A is about to start emitting non-compatible messages in the first place. This would mean that would need to maintain some matrix or something over which versions of message X is support by which consumer, and defer deploying service A (with the new version of message X) until all the consumer are ready for it. How to implement this effectively I don't know.

Other alternatives

How does other handle message type compatibility between services that communicate using MQ - how do you know that when your service A makes a change to message type X, it won't break any of the consumers?

PS. I posted this over at Reddit a few days ago, but due to the lack of feedback I decided to post here as well.

  • 1
    First of all, you definitely should keep some form of registry for managing which services communicate with each other over which queues/topics, which message format versions are accepted by consumers, which versions are used by producers. Whenever a new format version is introduced, all consumers must be ready to understand it before producers using it are deployed, with the possible exception of purely backward-compatible changes. – Hans-Martin Mosner Jul 8 at 7:21
  • 2
    Can you release a new version of the API on a new endpoint? – Tvde1 Jul 8 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.