What should we do in face of a failing sub in pub-sub?

Question

I'm working on a microservices application that is implemented from the ground up with MediatR em CQRS. We have a list of domain events that will be published via MediatR [simple pub-sub library that implements a mediator to commands and events] as soon as a Command (CQRS) commits the database.

Let's imagine a new user was created in our application through a Command (CQRS). After creating the resource, an event about this was published through the MediatR and we have 5 subscribers to this event.

What should happen if 1 of those 5 subscribers fail and the whole transaction needs to be undone? Please assume that I'm talking about domain failings, not stuff like unhandled exceptions. This failing subscriber can't proceed properly because its domain knowledge determined that (eg: a payment transaction could fail in a subscriber of a domain event because the user doesn't have enough money).

Actually in this project we are simply calling MediatR.Publish (an async method) in a fire-and-forget manner, without awaiting or checking the result of the task. I'm concerned that the application can enter in an inconsistent state if we have failing subscribers.

Answer 1

Having 5 subscribers for one event seems only appropriate if all 5 actions are independent, in that the failure of any one doesn't affect the others.

If the failure of one, A, affects another, B, then delay the execution of B, until the success of A is known.  So now instead of 5 subscribers to one event: 4 subscribers to that event and A generates a new event that B subscribes to instead of the original event.  If A fails, then B does not get an event.

Otherwise, if there is a circularity in failures, yet subscribers can tolerate temporary inconsistency (e.g. all that's needed to do is reclaim resources like a reservation) an undo or error event can be generated.

And still, if we cannot tolerate temporary inconsistencies and also if there is a circularity in failures affecting others then will need to set up a kind of commit/rollback capability, where each subscriber tentatively does its operation and reports by event whether it can succeed/fail, and some other handler is listening for either total success or partial failure, and informs the others of it to commit or rollback.  (If failure in the commit process itself is possible even after reporting success, then we in 2 phase commit territory.)

It would probably be best if to avoid the latter.

Answer 2

Reduce your scope for errors. Some of the things you said were errors, are not errors and should not be treated as errors. At least not in the context of creating a user.

Let's say you are creating a user, and you have to set their password, create a profile, and create a shopping cart.

It makes sense that the login service handles users and their passwords, so that is all synchronous within the same microservice. Possibly you use an SQL database and you insert the user with an initial password. It's not possible for the "set password" step to fail if the "create user" step succeeds because they are the same step. So far so good.

Now you send an event to the profile service to create a user profile - containing their display name, date of birth, etc. What if this fails? You still created a user account - but they have no profile. And you want to delete the user account. Should you?

Microservices are ideal when a different team works on each microservice, and that is the reason they were invented to begin with, following Conway's Law. If you are on the login team, you are not responsible for the errors of the user profile team. If you are on the user profile team, you wouldn't require the login team to provide additional API surface just to handle the cases when you fucked up - if you try to push that on them, they will rightfully point out that it's easier for you to not fuck up, than to add more untested API surface for error recovery, which is more likely to have errors (especially because you didn't test it) than the first error is to occur in the first place. What if the error recovery API fails? Will you add error recovery recovery recovery recovery recovery?

From the user profile service perspective, receiving these events from the login service is no different to receiving them from a third-party website. Imagine you have a bot that processes Twitter tweets. Twitter sends you an event saying "User @JohnDoe1234 tweeted: lol hi guys". And your software crashes. Do you send a message back to Twitter asking them to untweet the tweet? No, you have to accept that the tweet happened, and you have to accept that your software crashed. You get to decide what to happen from there. Maybe you ignore the tweet, or maybe you fix the software and then you process the tweet again with the new version of the software. One thing you absolutely can't do is prevent the tweet from happening. It already happened, it's done.

And if @JohnDoe1234 in the meantime comes by and asks your program to display his Twitter analytics, what should it display? Perhaps it pretends he never tweeted that, perhaps he gets a fail whale, perhaps he gets an ugly default ASPX error page. If it's his first tweet then presumably he gets a page saying "Sorry, we don't know that username."

And so it is for user profiles. A login account was created - you (as the user profile developer) can't undo that. If the user asks for their profile data you're just going to have to respond with an "unknown user account" error message. Meanwhile you are working to fix the software, and maybe next week you have fixed the software and processed the failed account again, and now it has a profile.

If you are using a message bus that records events, this may help you replay the failed event. If not, you may have to do it manually. You could even have a weekly scheduled task that looks at all the user accounts to make sure they have profiles. You shouldn't need that, but it's one possible thing you can do.

Microservices that display profile data should not rely on the profile service working. A distinctive feature of microservices is they have to be able to accept each other failing. The homepage should display "Hello, (error)! It's your (error) birthday today!" - or better, skip the hello message entirely - rather than crashing. It makes no difference to the homepage, whether the user profile service is just down today, or whether it's up but it doesn't have data for this user account even though it's supposed to. Either one should be logged, and investigated by the profile service team.

Same goes for shopping carts. Failed to create the user's shopping cart? They can still browse, until they add something to the cart, and the cart service sends an error ("unknown user"), and the user gets one of those frustratingly generic messages ("internal error! try again later") and maybe comes back later or maybe goes to a competitor. Still, if the user just wants to browse and not buy anything yet, they can. The loose coupling allows everything to work that's not related to the shopping cart.

Answer 3

Same thing you would do if you weren't using queues and distributed processing.

Construct a manual transaction where you do the five things, check at the end, undo as required.