21

My scenario is as follows.

I am designing a system designed to receive data from various types of sensors, and convert and then persist it to be used by various front-end and analytics services later.

I'm trying to design every service to be as independent as possible, but I'm having some trouble. The team has decided on a DTO we would like to use. The outward-facing services (sensor data recipients) will receive the data in their own unique way, then convert it to a JSON object (the DTO and send it off to the Message Broker. Consumers of the messages will then know exactly how to read the sensor data messages.

The problem is that I'm using the same DTO in a few different services. An update has to be implemented in multiple locations. Obviously, we've designed it in such a way that a few extra or missing fields in the DTO here and there are not much of an issue until the services have been updated, but it's still bugging me and makes me feel like I'm making a mistake. It could easily turn into a headache.

Am I going about architecting the system wrong? If not, what are some ways around this, or at least to ease my worries?

  • What kind of DTOs are you sharing, and what protocol are you using between services? It's okay to share, for instance, the proto file for gRPC or the avro schema for Kafka and generate the DTOs in both services, but I wouldn't share a shared library between two projects. – Vincent Savard Feb 20 '18 at 0:43
  • Encoded JSON strings and AMQP. I would prefer not to use anything language-specific. – Nickdb93 Feb 20 '18 at 3:03
30

My advice? Do not share these DTOs among the applications in any kind of library. Or at least don't do this right now.

I know, seems very counter-intuitive. You are duplicating code, right? But this is not a business rule, so you can be more flexible.

The service that sends the DTO needs to be rigid in his message contract, like an Rest API. The service can't change the DTO in a way that could break the other services that are already consuming the information from the DTO.

When a new field is added do DTO, you only update the other services that consume this DTO if they need the new field. Otherwise, forget it. Using JSON as content-type you have the flexibility to create and send new attributes without breaks the code of the services that don't map these new fields on his actual versions of DTO.

But if this situation is really bothering you, you can follow the Rule of Three:

There are two "Rules of Three" in reuse: (a) It is three times as difficult to build reusable components as single use components, and (b) a reusable component should be tried out in three different applications before it will be sufficiently general to accept into a reuse library.

So, try to wait a bit more before sharing this DTO among the services.

  • 1
    Very much appreciated. This is really one of very few major concerns I have going forward. Not enough to keep me up at night, but enough to worry me. – Nickdb93 Feb 20 '18 at 3:04
  • 4
    Duplicated DTOs (in different and very independent services) don't violate DRY. That's it. – Laiv Feb 20 '18 at 11:19
  • 3
    Presumably then there's no reason not to copy the DTO source code directly from one project to another, as a one-off operation, although then any parts not required in the new project should probably be deleted. – bdsl Feb 25 '18 at 17:44
  • 1
    Even a whole service might be delete causing no major problems to the whole system. Ideally. – Laiv Feb 25 '18 at 18:27
  • 4
    To clarify the essence of the answer, when developing microservices, each service should be developed as if they do not know about the other services, except for the actual contracts it may require. – Jonathan van de Veen Feb 26 '18 at 12:02
9

When it comes to Microservices, we often forget one important characteristic of the architecture. Services' SDLCs are (or should be) independent too.*.

Different SLDC and different dev teams

We should bear in mind that there could be more teams involved in the development of the MS ecosystem, each of which in charges of one or more services. In turn, they might be located in different offices, cities, countries, plan... Perhaps, they don't even know each other, making very hard for them to share knowledge or code. That could be very convient because shared coud also implies a sort of sharing reasoning and something important to recall is that, whatever makes sense for a specific team, doesn't has to make It for another team necessarily. For instance, the DTO's data structure could be convenient for one team (service) and not to be for others.

Different needs, different technologies

Isolated SLDC' also allows teams to choose the stack that best suits their needs. Imposing DTOs implemented in a specific technology limits the capacity of the teams for choosing the adequate tools and stacks.

DTOs are neither business rules nor services contracts

On the other hand. What are DTOs? Plain objects with no other goal than moving data from one side to another. Bags of getters and setters. It's not the kind of "knowledge" that worth reuse, overall because there's no knowledge at all and these are components prone to change with all that it means.

Contrary to what Dherik has stated, it must be possible for a service to change its DTOs without having to make other services to change at the same time. Services should be tolerant readers, tolerant writers and fail tolerant. Otherwise they cause coupling in such a way that makes the service architecture a no sense.

Different business, different interpretations

While there could be (and there will be) cross-cutting concepts among services, it does not mean we have to impose a canonical model to force all services to interpret them in the same way. Recall what I said about the DTOs data structures.

Case study

Say our company has three departments, Customer Service, Sales and Shipping. Say each of these releases one or more services.

Customer Service, due to its domain language, implements services around the concept of customers, where customers are persons. For instance, customers are modeled as name, last name, age, gender, email, phone, etc.

Now say, Sales and Shipping model their services according to their respective domain languages as well. In these languages, the concept customer appears too but with a subtle difference. To them, customers are not (necessarily) persons. For Sales, customers are a Document number a Credit Card and a billing address, for Shipping a full name and a shipping address too.

If we force Sales and Shipping to adopt the canonical data model of Customer Service, we are forcing them to deal with unnecessary data that could end up introducing unnecessary complexity if they have to maintain the whole representation and keep the customer data in sync with customer service.

Related links


* Here is where the strengths of this architecture lays on

4

I'm trying to design every service to be as independent as possible

You should be publishing events. Events are certain type of messages that represent a solid fact about something that has happened at a particular point in time.

Each service should have a very well defined responsibility, and should have the responsibility of publishing the events related to that responsibility.

Furthermore, you want your events to be represent business related events, not technical events. E.g. prefer OrderCancelled event to an OrderUpdated with status: "CANCELLED".

That way, when a service needs to react to a cancelled order, it just needs to listen to a that particular type of message, which is carrying only data relevant to that event. E.g. an OrderCancelled probably just needs an order_id. Whichever service that needs to react to this has already stored whatever it needs to know about the order in its own data store.

But if the service only had OrderUpdated events to listen to, then it would need to interpret on the flow of events, and it was now dependent on delivery order to conclude correctly when an order was cancelled.

In your case, however, as your are publishing sensor data, it could make sense to have a service, listen to the events and publish a new stream of "business events", e.g. TemperatureThresholdExceeded, TemperatureStabilised.

And be careful about creating too many microservices. Microservices can be a great way of encapsulating complexity, but if you don't discover suitable service boundaries, your complexity is in the service integration. And that is a nightmare to maintain.

It's better to have too few, too large services, than to have too many, too small services.

  • I definitely agree. The sensor data comes straight into a microservice which parses the message and transforms it into a format agreed to organization-wide before publishing it to the broker. We will have some services that read the message and persist it to a database, and others which perform analysis of the message and do their own thing with it. Each service, by nature of the business, doesn't have much to do, thus leading to (hopefully) pretty simple services. – Nickdb93 Feb 26 '18 at 13:14
  • 1
    @Nickdb93 - In your case, as your are publishing sensor data, it could make sense to have a service, listen to the events and publish a new stream of "business events", e.g. TemperatureThresholdExceeded, TemperatureStabilised. (added to answer) – Pete Feb 26 '18 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.