12

In several of our company's applications, we use a custom logger. It's fairly robust, though we may replace it with something like NLog in the future. One of the logger's tasks is to log any exceptions encountered in the application.

One concern I've always had is that the exception handling within the logger allows for a silent failure. That is, if the log isn't written for a given exception (due to an error in the logger), how should I handle it and (somehow) log the exception in the logger itself?

Let's say the WriteLog function throws an exception. Should I try to call the function some number of times or until the exception isn't thrown? Should I try to write the thrown exception with the logger (which would likely just result in exceptions all the way down. . .)? I have been lucky enough to not encounter this situation except when we were first implementing the custom logger. On the other hand, I have no way of knowing at the moment if the logger has failed to log application exceptions (due to its own exceptions).

I have tried searching online and on some SE sites, but it's been fruitless so far since all the posts deal with errors in a logger (but not potential exceptions and how to log them) or with exceptions outside the logger.

  • 2
  • 5
    Log to stderr that your output medium has failed or that the "impossible" happened. – Doval Dec 12 '14 at 17:34
  • 1
    Send an email to the developers or just display the error with an email address and let the user copy & paste the error. – Chloe Dec 12 '14 at 19:32
17

When you encounter exceptions within the logger itself, you shouldn't use the logger to log its own exceptions. The reason for that is that:

  • You may be stuck in an infinite loop. Imagine that within your logger, you have a conditional branch which wasn't tested (and generates an exception). Imagine that once the condition is met, any further reported exception is handled by the same branch. This means that from the moment the branch is executed, you're in an infinite loop.

  • You may be stuck in a temporary loop, generating thousands of exceptions per second. Imagine you're reporting exceptions to a remote server. An issue with the server causes another exception, which causes another one, and so on, until the connection is back.

What you should do instead is to fallback to a safer way to log the exceptions. For example, if your logger sends the exceptions to a remote server, send the exceptions within the logger to syslog instead. If your logger records exceptions in Windows Events and this action fails, store the failure exception in a simple text file.

Once you have that, the next question is how do you know that those exceptions occurred: if you have dozens of applications running on thousands of servers, you can't possibly SSH each of them on regular basis to check whether they were logging something locally.

One way is to have a cron job which checks for those “exceptional logs” and pushes them to the location where other exceptions are stored (eventually using your logger, but beware of infinite or temporary loops!).

  • I ran into this same problem with my exception logger that went to e-mail. If it failed to connect to a server, it got into a terrible infinite loop. So instead, I put a check in place to divert to the Event Log and prevent new e-mails from being sent out until a new connection could be made. – mgw854 Dec 12 '14 at 17:41
  • I think we will try to implement a fallback as you suggest. Jon Raynor's suggestion to stop the application (in a critical logging situation) is also one we may pursue that we hadn't considered. – Zairja Dec 12 '14 at 20:27
  • What if you end up with timeouts sending to syslog or I/O errors writing to a file? You could still be making the problem worse, if the failures are due to a congested network or running out of disk space. This isn't exactly a holistic solution; you need to consider the possibility that there may not be any safe way of logging the errors. It's not that dangerous to log to your own logger as long as you incorporate cycle detection, exponential back-off, etc. – Aaronaught Dec 13 '14 at 5:48
11

If logging is critical to your application, then one should stop the application if logging fails.

If not critical, then being somewhat defensive one could have a secondary component to handle logging failures that logs/alerts to a secondary source. But even that is not fool proof and you will have to consider what happens if the secondary logger fails while it is monitoring the primary logger.

A good strategy is logging to a local file and if that fails, maybe logging that failure to the event log, generating an email alert, saving to a database, etc. With the logging frameworks that are available this should be foolproof unless the machine runs out of disk space or some other rare condition.

Ideally your better off failing silently as that will make the application less complex.

More importantly, to handle logging failures one should be monitoring the logs from a 3rd party. Over time you should be able to discern how many events a healthly application is logging. If it starts logging low or no events, then through monitoring you can see the problem occurring and potentially alert through that 3rd party mechanism.

  • 1
    +1 for making the distinction between critical and non-critical logging, as well as noting the importance of the number of logs per lapse of time. I'm disappointed that I haven't thought about those two aspects, while I've been using fallback logging for years. – Arseni Mourzenko Dec 12 '14 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.