0

Currently, our architecture uses an "API-first" approach in building our product. This product is divided across multiple teams handling different microservices.

enter image description here

The above image succinctly shows our approach. There are JSON Schema files that are shared between all microservices. These JSON schema files are used to auto-generate DTO classes for every microservices. These JSON schema files are contracts between the services. RabbitMQ is the backend for all the microservices. The message is published or consumed using these DTO classes. The classes are serialized into JSON format and sent across RabbitMQ. While receiving, the JSON format is deserialized into DTO classes.

Since the underlining domain model does not directly map to the JSON Schema, there is a translation layer, which extracts relevant properties from DTO classes and fills the domain model for processing the business logic.

The problem we are facing currently is, if there is a change in JSON schema files, all the services using the files, along with the translation layer in the respective services need to be adapted to imbibe the change. Also, the translation layer is growing with every change to the JSON schema.

One possible solution here may be to create JSON schema files based on the domain model directly. Though this does not obviate the need for a translation layer, it would make it short enough to maintain. The new architecture is shown below.

enter image description here

The main problem here is, this is not an "API-first" approach. Two microservices having the same domain class might differ in meaning, eg. Person(age, height, address) and Person(strengths, weakness, insurities). In this case, a translation is not possible.

One possibility here is to somehow share the JSON schema across all the microservices, using a shared configuration service(like Netflix Archaius).

My questions are

  1. Is such an architecture feasible?
  2. If yes, how can I share JSON files(probably using a shared configuration)?
  3. If no, what alternative approach should I use?
3
  • Why do consumers need to be aware of the updates immediately? Why can't you change producers in a backwards compatible manner and then update consumers as and when the consumers need the new data? Jun 2 at 13:08
  • @user1937198 I agree that not all consumers need to know about the update, but if someone's domain logic depends upon the updated data, that it needs from other service(s), that/those service(s) are compelled to be updated. Also versioning, in order to ensure backward compatibility, would leave a lot of boiler plate code, targeted to handle the version difference.
    – Nishant123
    Jun 4 at 14:54
  • Versioning is not the only way to ensure backwards compatibility. If you configure your consumers correctly, adding a new field with a new name is a backwards compatible change. Jun 4 at 15:25
3

The problem we are facing currently is if there is a change in JSON schema files, all the services using the files, along with the translation layer in the respective services need to be adapted to imbibe the change

Microservices is meant to allow independent development of each microservice. So each microservice should be responsible for it's own API, including any schemas it exposes. Each microservice would be responsible to ensure backward compatibility according to whatever policy you have. This should ensure that services can be deployed according to different schedules without affecting other services. If schemas are shared between services without a clear owner or change management policy, it becomes difficult to claim that the microservices are actually independent, it might even be a distributed monolith.

Overall I would recommend carefully considering the advantages and disadvantages of each kind of architecture. Microservices have distinct advantages, but also disadvantages. It sounds a bit like you are trying to have the advantages of independent microservices while not actually committing and accepting the cost of potential code duplication and strict version/change management.

Edit:

It is perfectly acceptable to have different definitions of something depending on the context. A UI-service might have a line that is a pair of screen-coordinates and a colour, for a map-service it might be a sequence of geo-coordinates using some specific projection. If there are conflicts regarding what a schema should contain, chances are it is should not be a single thing, since it is obviously used in different contexts.

If two services need to agree on how to communicate the teams should talk to each other to decide on how something should be done, but the team that implements the service should also own the API and any associated schemas, have the final say in it's design, and have to support said API for however long you decide to.

Now, Designing a good API is very difficult since you need to balance ease of use, generality, flexibility, security, backward-compatibility etc. This typically requires experience, so microservices are often most successful when you already have a good understanding of how they should be designed.

8
  • I understand that everything in the microservices needs to be decentralized. But that doesn't work when 2 or more services need to talk to one another. Whenever a service needs to communicate with other services (or the outside world) about a Thing, it should use the authority-provided definition of that Thing. JSON schema files, in this case, provides this definition.
    – Nishant123
    Jun 2 at 13:01
  • 1
    @Nishant123 Yes, but there should be someone responsible for each schema file, and changes should be done carefully to ensure backwards compatibility, either by avoiding breaking schema-changes and/or maintaining multiple versions of schemas as needed. if that is done you should not have the problem you describe.
    – JonasH
    Jun 2 at 13:07
  • 3
    @Nishant123 Why do the services need a consistent idea of Thing? They should only care about the parts of Thing they are actively involved with, not everything the producer is emitting, which should allow you to add to Thing on the producer side without impacting consumers. And remove parts of thing by first removing all uses from the consumers and then updating what thing emits. Jun 2 at 13:26
  • @user1937198 > Why do the services need a consistent idea of Thing? In order to avoid multiple definition of the Thing. See the Person example I have mentioned in the post. The rest of the things you mentioned, is already being done in the transformer layer.
    – Nishant123
    Jun 2 at 14:47
  • 1
    @Nishant123 If the consistency Must be maintained, then you have tight coupling between your microservices, which means you lose many of the benefits such as independent deployment. So it is worth thinking about why you want this consistency, and would you be better off with an approach which doesn't give it to you. Jun 2 at 15:51
0

You have come to the realization that independence does not mean independence huh. You have "loosely coupled" services because of the message queue (at least I hope your services are not expecting responses from the messages that get sent to the queue), but services are still coupled, just the direction of the coupling changed (people like to skip this tiny detail). This does not mean that you have a distributed monolith, but it means that you are working with a system... which is more complex, because there is 4 different code bases in 4 different languages and out of your reach to change.

If I understand you correctly, you have Service 1 which emits {x, z, y} and Service 2, which consumes it and translates it into it's domain (which is a very good design). Service 1 is thus independent of Service 2 and Service 2 is also independent of Service 1.

BUT the translation logic is dependent on Service 1. So if you change Service 1 to emit {q, x, z}, Service 2 will fail, because it might have used y, which is no longer present.

You already have Service 1 responsible for the schema it emits, so this is basically the most you can do.

There simply isn't a good fix for this as far as I know, other than to not introduce breaking changes to your events - do not update field names and do not remove fields. There is a book by Greg Young, Versioning in an Event Sourced System, which might give you some ideas, even if it is not entirely related to the messaging.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.