Our group builds and is responsible for maintaining multiple apps for a larger institution. We are in the process of breaking up our monoliths into reusable microservices (hooray!). When we got up to notifications (primarily email notifications) we got stuck. Our users consume multiple applications, and each service may have dozens of trigger points which would fire off a notification to other users. Do we adopt the philosophy that every microservice should handle their own notifications or should all notifications belong to a service unto themselves and rely on database triggers?
I am looking for articles or perhaps some experience advice since this is something we really, really can't afford to get wrong.
The argument for notifications in service
Microservices should be self contained. Each is responsible for its own state, its own model and should should [as much as possible] not be dependent on another service to function. Notifications are an essential feature for many services. What good is a chat service if you have to open it up every time to check for new chats? On actions, the microservice should fire off notifications - whether directly or by enqueuing it with some mail delivery service. This also helps with mitigation and further development. While iterating on the service, the developer is always aware of what notifications may be sent out depending on changes to the application state.
The case for database triggers
Considering the volume of notifications being sent out, it is best to have them centralized. Any updates, additions or deletions of notifications should occur in a single place. Furthermore, having notifications sent from the controller (or actions in flux) is risky, since it is possible that after the action is taken place, the DML operation fails. This would result in a user receiving a notification which is out of sync with the source of truth - the database. By having the notification service solely rely on triggers in the database, it also keeps to service encapsulation. The notification service does not care which service updated which model, only that the model did indeed update. Another risk is multiple applications operating on the same table (while this may seem like poor architecture, you may have an admin app and a user app). If there are multiple services updating a table, you may have multiple notifications going off to users with conflicting entries. By centralizing it, you are able to ensure that only a single notification goes off (perhaps by throttling how many notifications go off in a single interval, thus only notifying the last of the changes).
We are responsible for quite a few microservices that each have dozens of notifications. Our team is growing quite quickly, and my fear is that we're going to have notifications all over the place. Developers may accidentally make a switch in code, not realizing that this is going to trigger a ton of notifications to some poor user. What is important to me is consistency through out the org.
I'm hoping for some advice on the tradeoffs between the two architectures described above - notifications from application logic or from database triggers.
Thank you so, so much for your help and consideration!