2

We have a bunch of microservices written in different programming languages. These all communicate with each other over RabbitMQ(AMQP) middleware. There are consumers and publishers in each microservice which listen to incomming messages and publish on a RabbitMQ exchange respectively. The current architecture is shown in the image below

enter image description here

I want to decouple the Rabbit specific consumers and publishers from the microservices and implement them in a separate shared library as follows.

enter image description here

This shared library shall implement RabbitMQ exchange bindings. The consumer in this shared library, should listen to any incomming messages and invoke the relevant method in the target language. Same for publishers, the target language should be able to invoke a method in the shared library responsible to publish message on RabbitMQ.

The problem I am facing is in making target language bindings with my shared library.

How should I share the functions of the target language that can be invoked by the shared library?

The reason for having a shared library is because, I can swap the current RabbitMQ message service with any other.

I request for some suggestions to approach this problem

EDIT

These microservices are written by a variety of teams spanning across different companies.

The reason for having a shared library

  1. Since my current services directly implement RabbitMQ bindings, the integration test cases for each of the services also have a dependency on RabbitMQ.

  2. I can easily swap RabbitMQ for any other messaging service(for example Kafka) without changing my services, as this makes the services oblivious of the underlining messaging service.

5
  • 2
    Since RabbitMQ already has bindings for each language - your shared library approach is not necessary. Simply define the API you want to restrict your microservices to (or what features you want to add) and then implement it for each language you need. Since your API will expose features common to all messaging systems only (since you want independence from a particular messaging system) it will in each language be simply a thin wrapper over RabbitMQ's binding. Easy to write, easy to test. In each language. – davidbak Mar 26 at 15:11
  • The goal of your separate shared library is mainly to get a kind of wrapper for any message queue services ? – Zilog80 Mar 26 at 18:37
  • @Zilog80: Yes excatly. I can swap out RabbitMQ for any other message queue service for instance. – Nishant123 Mar 29 at 6:29
  • @davidbak: Writing wrappers for each language would ultimately create dependency on each microservice. A shared library would decouple excatly that from my microservices implementations. Also I would be able to write test cases independent of any messaging service. – Nishant123 Mar 29 at 6:40
  • Perhaps I didn't understand your question. Sounds to me - and the drawing looks like to me - that you want a wrapper not around your microservices but around your messaging system. "So that you can swap the current RabbitMQ message service with any other." – davidbak Mar 29 at 16:04
4

How should I share the functions of the target language that can be invoked by the shared library?

Most languages/ecosystems have packages, and a system which manages those packages and the dependencies between them: pip for Python, npm for JavaScript, NuGet for C#, RubyGems for Ruby, etc.

You create your wrapper as a package, and you rely on it from every service which uses a given language.

enter image description here

Another approach would be to use a different technology, such as HTTP (including Server-sent events) as an intermediary between the services and RabbitMQ. This being said, you'll have the exact same problem you have now if one day, you decide to move from HTTP to something else.

I want to decouple the Rabbit specific consumers and publishers from the microservices

The services are already decoupled from RabbitMQ through the use of AMQP. Essentially, you can replace RabbitMQ at any moment by any other AMQP-enabled server (note that AMQP 0.9.1 and AMQP 1.0 are two very different beasts, however). There are risks that any abstraction you'll add above AMQP would be a leaky abstraction: that is, it will look like you could replace at any moment, painlessly, your message queue service by something else, such as Redis pub-sub, but this is only in appearance. In fact, such replacement would involve major changes in most services anyway, if they were designed with RabbitMQ model in mind.

This is very similar to what happens when somebody implements a data access layer in order to isolate the application from the underlying database. The application is still designed with the underlying database in mind, it is optimized for the particular database, it uses its specific features that other databases may be missing, or works around its weak points.

This means that once the business decides to replace Oracle by PostgreSQL, it can be done with just minor changes, but when the decision is to replace Oracle by MongoDB, you're going to spend the next six months changing a lot of code way beyond the original data access layer.

2
  • Thank you very much for your answer. Actually all the services that I have mentioned are developed by teams spanning across different companies. The shared library approach that I have mentioned is an attempt to tie those services together without major changes to the existing services. The wrapper approach you mentioned, is promising, but wouldn't that extra code layer on top of the each service, be an invitation to(and handling of) new potential bugs? – Nishant123 Mar 31 at 8:47
  • 1
    wouldn't that extra code layer on top of each service, be an invitation to(and handling of) new potential bugs? Look at it this way, the shared lib could be buggy too and then everybody is affected by the same bug and blocked by the same roadmap to the solution. While Arseni's suggestion might affect only some of the consumers, not all of them. It's also important to bear in mind that different teams, different needs. Your unique shared lib would be hit by many and different needs, pulling in different directions. Generic, unique silver-bullet solutions rarely end well. – Laiv Mar 31 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.