Notifications in a microservice architecture: In Application Logic vs. Database Triggers

Question

Context

Our group builds and is responsible for maintaining multiple apps for a larger institution. We are in the process of breaking up our monoliths into reusable microservices (hooray!). When we got up to notifications (primarily email notifications) we got stuck. Our users consume multiple applications, and each service may have dozens of trigger points which would fire off a notification to other users. Do we adopt the philosophy that every microservice should handle their own notifications or should all notifications belong to a service unto themselves and rely on database triggers?

I am looking for articles or perhaps some experience advice since this is something we really, really can't afford to get wrong.

The argument for notifications in service

Microservices should be self contained. Each is responsible for its own state, its own model and should should [as much as possible] not be dependent on another service to function. Notifications are an essential feature for many services. What good is a chat service if you have to open it up every time to check for new chats? On actions, the microservice should fire off notifications - whether directly or by enqueuing it with some mail delivery service. This also helps with mitigation and further development. While iterating on the service, the developer is always aware of what notifications may be sent out depending on changes to the application state.

The case for database triggers

Considering the volume of notifications being sent out, it is best to have them centralized. Any updates, additions or deletions of notifications should occur in a single place. Furthermore, having notifications sent from the controller (or actions in flux) is risky, since it is possible that after the action is taken place, the DML operation fails. This would result in a user receiving a notification which is out of sync with the source of truth - the database. By having the notification service solely rely on triggers in the database, it also keeps to service encapsulation. The notification service does not care which service updated which model, only that the model did indeed update. Another risk is multiple applications operating on the same table (while this may seem like poor architecture, you may have an admin app and a user app). If there are multiple services updating a table, you may have multiple notifications going off to users with conflicting entries. By centralizing it, you are able to ensure that only a single notification goes off (perhaps by throttling how many notifications go off in a single interval, thus only notifying the last of the changes).

Question

We are responsible for quite a few microservices that each have dozens of notifications. Our team is growing quite quickly, and my fear is that we're going to have notifications all over the place. Developers may accidentally make a switch in code, not realizing that this is going to trigger a ton of notifications to some poor user. What is important to me is consistency through out the org.

I'm hoping for some advice on the tradeoffs between the two architectures described above - notifications from application logic or from database triggers.

Thank you so, so much for your help and consideration!

Answer 1

Third option: Make notifications its own microservice. Each microservice can utilize its own internal rules for communicating to the notification microservice. The notification microservice can focus on executing its own internal set of rules for pushing notifications given any of the various validation rules that need to be checked before pushing.

Answer 2

Well, your microservice is basically part of a Service Layer, correct? Your clients will subscribe to event endpoints on that Service Layer; that will not change. So the only real question is "from where will your events originate?"

Database triggers are the wrong tool for this. Database triggers have a different purpose; if you want to eliminate your monolithic architecture, putting your event originations there will have the opposite effect.

Centralize your events in a single microservice instead, or simply let each microservice handle its own notifications.

This isn't about a "single source of truth." Every microservice gets its data from the database in the state that it's in at the moment that it is retrieved. Events are not special in that regard.