Using Exception Emails as a System Status Indicator

Question

I'm working through our bug list today and I'm trying to clear up/fix issues that we commonly receive exception emails for. Although all the exceptions (so far) are handled, some don't actually represent an error in the code. For instance, if a third party service goes down temporarily, our code fails, handles the exception and sends us an email.

This can be especially useful during business hours because it allows us to quickly see if something is down and react to it. However, there are also scheduled down times and the exception emails can quickly fill an inbox.

My question is, is there a best practice for gaining insight into your system status? Is an exception email as I've outlined here an acceptable practice or should exception notifications only ever be unexpected?

Answer 1

I don't think emails are the best way to report errors like this.

Email obviously doesn't guarantee fast delivery and you need to have someone on the end of those emails.

If you have too many error reports, then things get missed and you start to ignore them due to the low signal to noise ratio.

Best practise in this area would probably be to use some third party monitoring and alerting software. There are hundreds of these products on the market which will do a better job than most home grown solutions. Nagios is a good open source tool in this area.

Re exceptions, sorry for the pithy answer but I think exceptions should only be used for exceptional circumstances.

If you regularly have scheduled downtime on your partner systems, then that should probably be handled cleanly either in the code or via some process rather than allowing your application to start throwing exceptions.

Answer 2

An exception email would be reasonable depending, of course, on the system, the company, the urgency of the error, etc. I would suggest that the error also get written to some log (I would prefer an error table in a database assuming the database is available but a log file would be perfectly acceptable) and that the decision about whether to send the email involve looking at that log file. If you encountered a particular error and sent an email 5 seconds ago, it's unlikely that sending another one is going to be helpful-- you probably don't want to send emails more than every few minutes for any particular error (possibly adding a counter to the emails to tell you how many times it happened since the last email). That prevents you from flooding an inbox with thousands of duplicate messages.

The larger the organization and the more critical the application, the more likely that you'd want to build some sort of status dashboard that someone could monitor and the more likely that you'd want to expose monitoring information in a way that a centralized piece of enterprise monitoring software can handle. If you've got a team of folks wearing pagers and watching dozens or hundreds of enterprise applications, getting thousands of emails from hundreds of sources is going to be slow, inefficient, and error prone. If you've got a small team of developers that also happen to be responsible for troubleshooting something if it happens to go wrong, email alerts may be an appropriate trade-off.

Answer 3

Hopefully you have a few customers that are paying a lot for this service. I'm working with an app that determines how to handle error notifications in a config file. Each error type is flagged to email or not and to what address(s). Some even email on success. Everything is logged in a file for each client app (errors could occur here and never make it to the server) and the app server.

For a service that is out of order, the client should know about it or at least on of their admins (email?), but I don't know if the developers/support staff need to.

As different companies mature with the system, many of these notifications are turned off or limited to a customers's own support email address.