0

I have been designing an email alert system for my customers which allows them to choose what alerts they receive. AlertManager seems suitable, except for it's static configuration file; A customer subscribing to an alert will require an update of the configuration file.

It seems like a hack to dynamically rewrite a configuration file. Every update risks crashing the program, and the config duplicates database information which can go stale if I'm not careful.

On the other hand, Dynamic Configuration also seems much more pragmatic. There's much less burden to generating a config file than handling the complexities of email - many of which I won't know about or consider in my own code. Note that AlertManager supports reloading the configuration in-situ with kill -HUP.

The two options I can see:

  1. Manage the dynamic configuration by encapsulating AlertManager in an API service, whose primary purpose is to handle the risk of dynamic config. (e.g. Automatic rollbacks)
  2. Write a separate "customer-mailer" service that accepts AlertManager's webhooks, and uses a database or API to find subscribed customers and send each of them the alert.

Question

I'm interested what option you think I should choose. What factors into your reasoning? Perhaps there is a more concise trade-off decision here.

I couldn't find any content on the internet on this topic yet I've come across this question a few times in my young career - dynamic nginx config being the classic one. A blog post discussing this type of decision would be excellent.

Thanks, I'm looking forward to your answers!

2

I would use a layer of indirection - send the alerts to a mailing list or non-user mailbox, manage the actual recipients with a mailing list manager.

Looks like GNU mailman has a REST API you could use.

Another option is to use a database as the source of truth for alert subscriptions, then use a templating engine to generate the yaml. Looks like there's a few yaml templating engines our there already. This would reduce the risk of accidentally rewriting non-subscription parts of the yaml.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.