How are new bounded contexts deployed, and bugs fixed, in message-driven "autonomous" microservice architectures?

Question

I'm coming from the monolith background, using a single large relational database. From my research, many proponents of the microservice architecture favor the event-driven rather than REST driven architecture. The following question applies to inter-service communication, between bounded contexts (in DDD terms) and NOT to events within a BC, nor event sourcing.

As I understand it, to decouple services from each other, a message bus such as RabbitMQ is used to publish a domain event, which contains info related to the event, such as UserCreated or UserSuspended, and other bounded contexts can make use of it to store the data they need in their own database. For example, a billing service might need just user id and status so that it doesn't generate invoices for suspended users. With this "cached" user data, it doesn't have to do a REST API call to the User BC to get the information. This is considered "autonomous" rather than "authority".

A couple questions:

1. How can new services be deployed? In the above example, assume the User service exists but the Billing service is being built. It needs to seed its database with all the existing user statuses. Assuming I don't have the entire history of events available in the User service (as with RabbitMQ), this means that one of two options need to happen: A) the User service needs to provide an API endpoint that the new Billing service can use in its migrations to get user statuses. Additional work needs to be done in the User service if this endpoint doesn't exist. Or B), the Billing deployment/migration script can break responsibilities one time and directly get access to the User database. But this means devs from Billing need to learn the data schema from another service.
2. How can bugs that cause inconsistencies be fixed? For example, somehow a developer broke the publisher and no UserCreated or UserSuspended events get produced for an hour. The bug is noticed and fixed, but a whole hour's worth of events are missing. Two possible solutions: A) the dev from User BC manually publishes the missing events. He would need to figure out which events were missed (if even possible), then write scripts to construct them (time consuming.) This would be additionally problematic if new events already fired that made the original missing ones obsolete. Or B) the other bounded contexts could be notified to run scripts that update their data via the User's API (returning to the authority model)

Another underlying question here is, in inter-service communication, where is the ultimate source of truth for external events? Is it in the event stream, or in the API provided by the bounded contexts?

I love the idea of autonomous systems to reduce coupling, remove dependencies, and to let services evolve faster, but I'm afraid of situations that cause inconsistencies that wouldn't happen in the authoritative structure, where data is always up-to-date via live API calls. Any tips would be greatly appreciated.

Answer 1

One approach is to treat notifications (messages pushed to a consumer) as a performance optimization, rather than as the only communication channel.

Greg Young's talk on Polyglot data (2014) describes this idea, starting from about 25 minutes into the talk.

the User service needs to provide an API endpoint that the new Billing service can use in its migrations to get user statuses.

This is the right idea. More precisely, we need endpoints and messages that describe both the query and the response. Some care is to be had in defining both, as the two ends of the conversation are separately deployable -- if you need to support the evolution of the messages over time, then you will occasionally be in situations where the deployments don't agree on which version of the message schema is in use.

the other bounded contexts could be notified to run scripts that update their data via the User's API

This is the approach that I would expect. You wouldn't necessarily need to "notify" the subscribers, if the producer just makes the "new" events available at the endpoints in the usual way.

A common solution is that the endpoint represents an abstraction of an append only stream, and each consumer keeps track of the latest position in the stream that it has processed. See Twitter's guide to Working with Timelines

Answer 2

In general terms, BCs/services are autonomous but not individualists. They collaborate to fulfill a larger goal. Therefore, as a general rule, asking another BCs developers to provide something is not a bad practice, in my opinion. This could be asking for an event to be published (UserSuspended might not be published at all, so you need to ask the developers of Customer Care service to publish it) or for a deployment functionality to be provided.

Answering your questions:

1. "How can new services be deployed?" Create a deploy only solution for this scenario. Don't let Billing mess with the other service data. Ask the other service to implement a well define interface, like IProvideSuspendedUsers. Create a deployment procedure for Billing that calls that interface and when done, remove both sides of it from the system (otherwise it'll become something to be maintained forever that nobody will know why it was created in the first place).

2. This could be trickier and it will depend on the situation, but I'd say that you event consumers should always be prepared to receive late and out of order events and also the developers of the faulty BC won't know what dependencies that event has, so they should, in my opinion, publish the late events and that's it (I would probably add sending an internal notification to all teams about this fact, just in case).

Answer 3

Sorry for my glib answer. Event Sourcing (ES) is already being used, it seems.

So to your first question, how can new services be deployed? The answer is that it depends (as with all things) on the new service. So there isn't a one-off answer that will apply. Using your Billing service as the example raises a couple of questions. First, what does the billing service do? Does it run on a periodic schedule? Does it need to know about events from other BCs in real-time? Let's say that you have a service that used to be free but now is going to paid-for, or have some paid-for features. Or perhaps you just want to start accounting usage internally. In that scenario, the billing service doesn't really need to know about every User, they only need to know about Users that are doing billable things after the billing service comes online. So you don't need to pre-populate the service at all. Without knowing more about your example, you're not going to get a good answer here.

To the second question, how can inconsistencies be fixed, does imply the use of some sort of ES at some point earlier in the system. If you don't have any ES at all then you have lost the original events and you have to, as you suggest, bring the state of the non-originating BCs into sync with the originating BC.

You are up against CAP. New events could be coming into the system as the state is being converged if you choose to keep it online. So if you want to bring the state into sync, you have to decide on whether or not you want to stop processing events (Availability) or just have inconsistent data in the non-originating BCs.

The ultimate source of truth is up to your design. I don't know enough about what you're actually trying to do in order to provide any more detailed answers. Sorry.