5

We have multiple microservices communicating over MQ. As MQ messages are the interface/contract between the services, whenever we make changes to the MQ message published by a service we need to make the same adjustments on the services which consume the message.

As of now, the number of services is small enough so that we know which services communicate with each other, and can keep the MQ message contract in sync between them. But as the number of services grow this becomes harder.

Option 1: Break things first, then fix it

I've been thinking maybe of implementing some kind of health check. Let's say service A during operations may emit message type X, which is consumed by service B. Service A could then on startup emit a health check type of message, something in the lines of a message X dry-run. When service B receives this, it simply verifies that the message is according to contract. If not, for example if service A have remove a critical field in the message, then service B will reject the message which in turn will end up on a dead-letter exchange, which again will trigger a warning notification to the devops staff.

This approach won't prevent us from deploying non-compatible message types, but will notify us pretty much instantly when we do. For our use case, this might work due to our very small number of developers and projects, so if we break things like this we'd be able to fix it quite quickly.

Option 2: Early probes

A variation over this might be that we start versioning the MQ message format (which we probably should and will do anyway). Then, when service A plans to upgrade from version 1 of message type X to version 2, server A could early on start emitting "dry-run" type of version 2 of message type X. This would cause service B to drop the message. Say this happens a few days or weeks before service A perform the actual switch from version 1 to version 2, then the devops team will have time to add support for version 2 in the mean time.

Option 3: Manually detecting conflicts before deployment

Another approach would be to have some way of detecting - before the actual deployment - that service A is about to start emitting non-compatible messages in the first place. This would mean that would need to maintain some matrix or something over which versions of message X is support by which consumer, and defer deploying service A (with the new version of message X) until all the consumer are ready for it. How to implement this effectively I don't know.

Other alternatives

How does other handle message type compatibility between services that communicate using MQ - how do you know that when your service A makes a change to message type X, it won't break any of the consumers?

PS. I posted this over at Reddit a few days ago, but due to the lack of feedback I decided to post here as well.

3
  • 1
    First of all, you definitely should keep some form of registry for managing which services communicate with each other over which queues/topics, which message format versions are accepted by consumers, which versions are used by producers. Whenever a new format version is introduced, all consumers must be ready to understand it before producers using it are deployed, with the possible exception of purely backward-compatible changes.
    – Hans-Martin Mosner
    Commented Jul 8, 2019 at 7:21
  • 2
    Can you release a new version of the API on a new endpoint?
    – Tvde1
    Commented Jul 8, 2019 at 9:38
  • If you don't version the API then every time I've seen this each and every message contains a version # field (for that message type) and the services need to handle back versions (to some limit). This is especially the case if any of those communicating services don't belong directly to you - e..g, they belong to either a different company or to a different team in the same company under a different budget structure. (Because their goals - therefore their planning, and their timeframes - are different from yours.)
    – davidbak
    Commented Jun 10, 2021 at 15:51

1 Answer 1

Reset to default
1

Disclaimer: I have not done this in practice, but have had some (probably too much) time to watch talks and read books on architecture. I have been working as a dev for 6 months on a monolithic backend app solo. I'll just throw some things I have read/heard about messages.

You can view your service in two ways as far as I can see. It is either an independent thing (as in it doesn't care about the world, the world has to adapt to it), or you can view it a thing that has contracts to fullfil.

I think that the appeal for microservices comes from the first interpretation. Lets say you have services A and B. A is the producer, B is consumer. If you view it like that, then you can create "Contract Tests" on the consumer B, which validates the integration with the service A. These tests are basically integration tests, where you make service A produce a message and you validate that the message delivered to B was correct.

The second one is trickier, because you must move these tests from service B to service A. Basically you now have all of the contracts on service A as unit tests, which sounds kind of good to me, but the problem with that is that A is now dependent on B instead of the other way around.

I would try a different thing though which might indeed be stupid, but as of now I cannot see how:

You could try having a message module where you define what your messages look like or even better for me would be a client that the service A must provide and is responsible for testing and modifying. Service A could also be responsible for maintaining test doubles, making "integration tests" easier and faster, but kind of trading off the test independence. I don't see it as a big deal though, as this is all internal code to the company (I would rather have tests blow up early).

You would then wrap the client with an adapter on service B and transform the service A messages to service B objects, maintaining B's independence. Whenever an update happened, you would get errors at compile time (assuming you are using a strongly typed language).

You would however still have to somehow resolve the deployment issue, which might happen if you have different version of services running at the same time, but you can never escape that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.