Boneheaded exceptions should not be caught. Then how to provide fault tolerance and reliability?

Question

I've always been taught that fatal exceptions (indicating problems that cannot be solved programmaticaly) and boneheaded exceptions (resulting from bugs in my code) should not be caught, should not be handled, should not be ignored, instead they should be allowed to crash the app. The reasoning is that once such an exception is thrown the state of the application is undefined so we need to stop any further damage that might result from keeping our app running. Also bugs should be fixed and crashing the app forces us to do so ASAP; if the app ignored the exception instead then the bug could remain unnoticed and unfixed.

Boneheaded exceptions are your own darn fault, you could have prevented them and therefore they are bugs in your code. You should not catch them; doing so is hiding a bug in your code. Rather, you should write your code so that the exception cannot possibly happen in the first place, and therefore does not need to be caught. That argument is null, that typecast is bad, that index is out of range, you’re trying to divide by zero – these are all problems that you could have prevented very easily in the first place, so prevent the mess in the first place rather than trying to clean it up.

(https://ericlippert.com/2008/09/10/vexing-exceptions/)

However, there are cases where crashing the application seems unacceptable.

Let me provide an example. We need to synchronize data between an e-commerce website and an ERP system. To this end we have a service that periodically downloads new orders from the e-commerce platform and inserterts them into the ERP system. Both e-commerce shop and ERP are 3rd party solutions that are outside of our control.

This process can fail for numerous reasons:

• The ERP system is a poorly documented mess. It can reject any data that fails its validation rules (such as: surname too long) and unfortunately its validation rules are not immediately clear. I also saw cases where calling setters exposed by the public API of the ERP in the wrong order produces exceptions. As well as a plethora of other not immediately obvious cases where ERP might throw.
• The e-commerce system returns orders serialized to JSON. These JSONs will have to be deserialized by our app. Unfortunately the exact fields of the JSON may vary slightly depending on the details of the order and it does sometimes happen that the deserialization of a particular order fails because we did not foresee some particular condition.

This does not invalidate the general rule that boneheaded exceptions should not happen in the first place, but it does make it rather hard to ensure that no boneheaded exceptions may ever happen.

But now requirements of high fault tolerance come. It is unacceptable to fail to transfer all orders because a single order could not be transferred.

This leads to following pattern:

for(var order in ordersToTranfer)
{
    try
    {
        WriteToERP(order);
    }
    catch(Exception e) // any exception
    {
        LogException(e);
        AddToListOfErrorsThatRequireRetryingAndPossiblyManualIntervention(order);
        continue;
    }
}

This violates the general rule that boneheaded exceptions must not be caught.

Is there any way to have the cake and eat it too? To avoid catching all exceptions while still stopping the failure of transferring a single order from crashing the whole application?

All that comes to my mind is to move from a monolith to microservices. Service A downloads orders from e-commerce website but does not deserialize the JSON except insofar as necessary to split the list of orders into individual orders. Then it invokes an enpoint exposed by service B once for every order. Service B deserializes each received order and passes it to service C (again invoking an endpoint of service C once for each received order) which saves the order to the ERP. If the services are implemented as, for example, stateless HTTP servers that expose a REST API then fault tolerance will be provided by the environment? A typical server does not stop processing other requests because one request faulted; instead it returns HTTP 500 from that particular request and handles other requests without interruption? Solutions such as IIS or Apache enforce this.

• Is the requirement of high fault tolerance enough of a reason to pay the non-trivial costs of increased complexity that come with the microservice architecture?
• If yes, then what are the exact advantages of doing so as opposed to manually catching all exceptions?
• If no, then what to do instead?

Answer 1

Fail as early as possible, and catch in context.

Going by the definition on https://ericlippert.com/2008/09/10/vexing-exceptions/, a boneheaded exception isn't one that should not be caught, it's in fact one that should not be thrown - because the underlying problem can be solved elsewhere.

For example; don't rely on a DivisionByZeroException; validate the input before you divide by it; after all, you decide if the division should take place in the first place. If the input is all messed up, you're missing a proper validator checking values as they come in.

In your case, this translates to don't rely on the ERP system to validate and report on errors you can prevent. Do not call WriteToERP(order) if you know it will fail. For example, when order is NULL. Also, don't blindly trust the response JSON; writing a validator and having it throw a BadResponseException is a lot better than catching any and all NullPointerException instances, making assumptions about where they come from and why a value might be NULL.

If you can't fail early, because the system is liable to change without notice, or you simply don't have the knowledge to properly validate your requests, there is nothing wrong with a more generic exception - just make sure to throw an exception as early as possible, instead of having known errors propagate down deeper into the process.

Answer 2

Your examples are both in the area of interfaces to systems that are not under your control, which is different from the interfaces between components that you control and where you can ensure that the calling side respects the interface specification.

You don't need to rewrite your system into a microservice architecture just to handle errors at a finer granularity. It is entirely reasonable to process a list of orders, checking each for correctness according to your understanding of the interface spec, and reject (or save for further inspection) those that don't meet the specs. There's no need to crash the whole process, especially when you cleanly separate checking of inputs from processing, so your application did not modify anything yet when it finds a problem.

Answer 3

"Fail fast" is a good default, but fault tolerance may be worth it in some cases. You just have to be really careful how you do it, because an unexpected exception means some of the program state is corrupted, and if you just ignore the problem you may propagate a corrupt state, creating even bigger problems down the line.

The rule is to isolate operations into independent units with no shared state. For example, if you post a set of orders to an external system, and each order is fully independent, then you can catch exceptions in one order placement, and then continue with the next (like in your code example).

Forget about microservices, that does not solve anything. Your thinking is correct though, since REST requests are typically processed independently of each other, i.e. a failure in one REST request will not affects other requests. But you can achieve the same just with ordinary functions or methods, as long as you have no shared mutable state. Depending on your language, this means avoiding global variables or static fields and avoid sharing mutable objects.

Answer 4

It's okay to sandbox (narrowly as possible)

While your ERP does indeed seem boneheaded, it is not boneheaded to sandbox a third party interaction. Just make it as narrow as possible.

Also, it would be prudent to throw an exception after a certain number of ERP errors occur-- this will save you from a runaway process, e.g. if literally everything is failing.

bool TryWriteToErp(Order order)
{
    //Do any preprocessing outside of the try block
    PrevalidateOrder(order);

    //Wrap only the third party interaction, and handle the fewest types of exceptions possible
    try
    {
        var response = _erpService.Send(order);
    }
    catch (Exception e ) when (e is SystemException || e is IOException || e is NetworkException)
    {
        throw; //Actual boneheaded exception; do not catch!
    }
    catch (Exception exception)
    {   
        RecordError(exception, order);
        return false;
    }

    //Do any postprocessing outside of the try block
    RecordCompletion(order);
    return true;
}


//Main program        
int errorCount = 0;
for(var order in ordersToTranfer)
{
    var ok = TryWriteToERP(order);
    if (!ok)
    {
        errorCount++;
        if (errorCount > _configuration.MaxAllowedErrors)
        {
            throw new CustomException("Too many failed orders.");
        }
    }
}

Answer 5

In the net, there are too many articles with advice on how to handle various types of exceptions. Even the wording "to handle an exception" puts the focus on the exception object instead of the method that failed.

But exceptions are the vehicle for a method to communicate its failure to its caller, with the details of the exception object describing the reason for the failure. In real life, nobody is interested in your reasons or excuses for failure, it just matters that you failed. And the same is generally true in software: important is the fact that some part of your algorithm failed, and not why it failed.

So, I can't agree with Eric Lippert here.

Boneheaded exceptions are your own darn fault, you could have prevented them and therefore they are bugs in your code. You should not catch them; doing so is hiding a bug in your code.

To me that reads as "make bugs in your code completely crash the application". I can't see that as a desirable system property.

Instead, I'd aim for

Make bugs in your code communicate failure in an orderly way to your caller, make sure they get logged with ERROR level, and prepare your system to accept the next call (that hopefully won't hit the same bug).

So, your code pattern looks completely valid to me.

It does not matter why some of your internal calls failed (boneheaded, vexing, exogenous or fatal exception, to use Eric's wording), as long as there's a chance to get the system back working for the next call.

In designing the correct exception handling, a few questions must be answered:

1. Can my method finish with success even though one of my internal calls failed? If yes, it's okay to catch exceptions here. Maybe, the catch clause has to take some action like using an alternative solution for the failed call, to allow success of my method. And typically, my catch clause should log a warning, unless we have some kind of "vexing" exception.
2. Will my method leave the system in an inconsistent state if aborted somewhere in the middle of execution? This can either be tackled by having a catch or finally clause that cleans up the state changes, or (often much easier) by re-organizing the method to first compute everything necessary for the state changes, and only then execute the changes in a near-atomic manner (making intermediate failures unlikely). Anyway, make sure the failure gets recorded in the logs.
3. Is there a chance to continue with the next call? Fatal exceptions typically tell you that the runtime system got a severe problem, so that an orderly execution of code can no longer be guaranteed. In such a case it's correct to shut your application down, and it's likely that you don't even get a chance to execute any kind of cleanup code before shutdown. So, generally it's correct to let fatal exceptions crash your application. But even that isn't a fixed rule. E.g., depending on circumstances, it can even be possible to recover from Java OutOfMemoryErrors.

With all these questions, most of Eric's exception categories are irrelevant. Regarding try/catch, you should treat exogenous, boneheaded, vexing and fatal exceptions the same. Only, vexing exceptions tend to give "true" as answer to the first question, and fatal exceptions a "false" to the third one.

Of course, becoming fault-tolerant doesn't mean you should stop fixing bugs now. You should still have your various levels of testing before delivering a system, and you should get and analyze the logs of production systems.

And then, regarding the developer action you should take, Erics distinctions make sense:

• Boneheaded exceptions are those that can be avoided by improving the code. Do that.
• Vexing exceptions are those that can't be avoided, but don't make any relevant method fail. So, finding such a situation in the logs means you got the answer to the first question wrong, and you should place a non-logging try-catch block there.
• Exogenous exceptions are those outside of your control, the only thing you can sometimes do is inform a system administrator to look at the missing file or access right or whatever.
• Fatal exceptions are those that trigger a chain of events with the result that the system is no longer able to recover. If you find such a pattern in the logs, it means you either brought the system into an inconsistent state earlier (possibly a wrong answer to the second question) or you got a fatal exception from the runtime system. You should at least detect the non-recoverable state and stop the system. And if the situation can be explained by an inconsistent state resulting from an earlier problem, locate and repair that.

Answer 6

There is no contradiction. Your example is a complex multi-stage system. You can still and should still throw within your part of the system if something unexpected happens and let it fail. It is up to the caller of your part to stop the process, to just ignore, to retry or to delegate to a mirror system. You are not to make your failing code succeed after all, that makes no sense, it is literally not your call.

Fault tolerance is not to be implemented by the failing part. It may be implemented by the user of your part.

Answer 7

Fault tolerance isn't created by you adding try catch blocks everywhere. It's part of the design.

For example, if you catch the "bonehead" should-never-happen ERP error and perform some logic, what if you made a similar error in your catch code? its catches all the way down!

There's no point in trying to second guess yourself like this, write tests and if they pass the code is good, if the fail, add more code.

Rather you would add a logging layer to your design, say pushing the message through a queue, which would keep the failed messages. Now regardless of how bad your code is, any failed message sent to the ERP can be reported and resent later.

You are right that with async calls in microservices, you simply cant know whether that message you pushed off into the system erred or not. So you have to have the fault tolerance built into the message transfer mechanism itself, rather than wrap everything in home rolled fault tollerance.

Answer 8

There are two questions

1. The specific situation with the ERP system
2. A generic question about exceptions.

For Q(1) about the ERP system: I don't think you need multiple services. Rather, you could pull the data into a series of staging tables, and add a "processing_status" column and an "error" column into those staging tables to document the success / failure for each record.

E.g. The first staging table could have the raw JSON the second staging table would have the deserialized JSON.

So the "processing_status" of the first Staging table would indicate whether JSON deserialization failed. The "processing_status" of the second staging table would indicate whether the ERP system accepted the data.

Next, Q(2) - This is about the Atomicity of an operation. I tie my Exceptions to the smallest possible unit of atomicity.

E.g. Say I'm pulling 10000 records from a source system. If the Sequence of these records matters, then a failure in processing any one of them should trip a rollback for the entire batch.

However, if these records are independent of each other, then my exceptions would be scoped to the function that processes a single record. If there was a processing error, I would set the "processing_status" and then save the exception details to the "error" column for that record. Then, a single alert could be raised at the end of the processing with a count of the number of failed records.

Exceptions are expensive to setup and teardown, and code that uses exceptions is always very readable. If there's a way to pre-validate with 100% assurance, I would use that instead of setting up an Exception. But when we are dealing with code written by others, or when the system has many moving parts, we do not have that luxury and that's when we have to finally rely on Exceptions.

The final aspect is graceful handling of errors. If our operating systems or browsers were to shut down for every error we would really hate it. These are complex pieces of code but they are usable because they handle errors gracefully and in some cases, also allow users to report problems to the code author.

So in summary, when I use exceptions, I scope them to the "Atomicity" of the operation, and then I try to handle the exception gracefully so that the code can process as much as it can without grinding to a halt.

Answer 9

Dealing with the ERP: it emits "boneheaded" exceptions and you can't fix it. Therefore it's acceptable to wrap it, although this should be done as close to its layer as possible.

Unfortunately the exact fields of the JSON may vary slightly depending on the details of the order and it does sometimes happen that the deserialization of a particular order fails because we did not foresee some particular condition.

This I think is much more addressable. There are ways you can address deserialization that either turn failure into a specific type of exception for "badly formed JSON file", or return error codes rather than exceptions. You can then "fuzz" your system with semi-random malformed JSON to test whether it throws exceptions.

Answer 10

Fail fast fail early in 2008 is a motto for developing single applications during that time. Or rather it works during development if you can control all the variations. In a fairly interconnected system, where different applications interact, the inputs and outputs cannot be sanitised in a way that bonehead exceptions are unhandled and the fix is to patch the root cause.

In this case, i'd offer the 'highlight and repair' approach to dealing with invalid inputs, which is to point out where the errors are, and either have them fix it on their part or include exceptions on yours. There's also a chance that errors on their system will not be fixed because of legacy issues and other downstream dependencies.

At this point the answer is less programmatic but more of a social one, which is to log enough of the errors, and if it impacts the running of the system, provide details for the other party to work on.