I have been designing an email alert system for my customers which allows them to choose what alerts they receive. AlertManager seems suitable, except for it's static configuration file; A customer subscribing to an alert will require an update of the configuration file.
It seems like a hack to dynamically rewrite a configuration file. Every update risks crashing the program, and the config duplicates database information which can go stale if I'm not careful.
On the other hand, Dynamic Configuration also seems much more pragmatic. There's much less burden to generating a config file than handling the complexities of email - many of which I won't know about or consider in my own code. Note that AlertManager supports reloading the configuration in-situ with
The two options I can see:
- Manage the dynamic configuration by encapsulating AlertManager in an API service, whose primary purpose is to handle the risk of dynamic config. (e.g. Automatic rollbacks)
- Write a separate "customer-mailer" service that accepts AlertManager's webhooks, and uses a database or API to find subscribed customers and send each of them the alert.
I'm interested what option you think I should choose. What factors into your reasoning? Perhaps there is a more concise trade-off decision here.
I couldn't find any content on the internet on this topic yet I've come across this question a few times in my young career - dynamic nginx config being the classic one. A blog post discussing this type of decision would be excellent.
Thanks, I'm looking forward to your answers!