RabbitMQ as backbone for services written in different languages

Question

I'm seeking advice from people using RabbitMQ to connect services written in different languages. Currently my organization is using RabbitMQ exclusively with C# and it's all working quite well. In order to provide some nice additional functionality over and above what RabbitMQ provides, we've implemented MassTransit. MassTransit provides some great benefits over RabbitMQ (abstracts away RabbitMQ specifics, handles "poison" messages, retries, serialization, provides lots of messaging patterns, etc.). However, the major downside is that it locks you into using .NET; it handles deserialization into .NET classes ONLY. This limits our ability to create services written in non-.NET languages. Of course we could surface RESTful endpoints written in C# to allow other services to connect, but that's not ideal as other services should be able to engage in pub/sub and other messaging not suitable over REST.

So here's the question: if you're currently using RabbitMQ, how are you connecting up services written in different languages? Do you have to implement some shared code for each language you want to support in order to handle serialization/deserialization?

Thank you for your time in reviewing this and sharing your advice.

Answer 1

You do it the same way as if you were calling the systems directly through a certain API, such as connecting microservices directly. You need a unified format.

A lot of, if not most, APIs have adopted JSON as their exchange format, because it's easily understandable and processable by pretty much any standard language today.

With RabbitMQ it should be the same, instead of serialising your entire .NET classes, you should serialise them into e.g. JSON and have a deserialisation logic on the other side.

In most languages serialisation to and from JSON is incredibly easy, there are numerous libraries (such as GSON for Java, Newtonsoft for C#, built-in methods in PHP,...). You will just need a bunch of new DTOs on each side of the message bus.

If you find that the JSON format is lacking something, perhaps it's too large, you could have a look at something like Protocol Buffers.

Answer 2

I've done what you're doing with sensor data. We've got lots of different types of sensors. Some include a lot of types of data, some very little, and often the data is in different units of measurement (meters vs feet, etc).

The team got together and created a JSON representation of how we would like the message to look when our consumers receive it from RabbitMQ. Obviously, some fields will he empty or non-existent, and some messages may have extra fields, but the first- and second-tier JSON objects in the message will be consistent.

Producers serialize their message into a UTF-8 byte encoded JSON object very close to the model we've designed, and send it to the RabbitMQ. Consumers deserialize it into a POCO/POJO what-have-you and do what they need.

At the end of the day, most languages can convert the message to a string and read the JSON object so we are comfortable using any language we wish.