How to E2E test handling of an unknown error in a client-server application?

Question

Using a ton of libraries, IO, ... you cannot handle all imaginary errors. You will need some form of "catch uncaught exception" handling to still have a control flow for such a case. But if you are aware of such an error, you usually can and want to add a proper handling.

That brings up the paradoxical question of how one would go about testing the handling of an unknown error.

Having a test case for the handling of an unknown error would make it a known error and would trigger adding appropriate handling, which in turn will invalidate the test case.

How should we test the handling of an unknown error?

One thought process was to specifical implement a method which can be triggered by some internally known condition which will raise an "unknown exception". But this feels wrong on so many levels.

Any other ideas?

Answer 1

That brings up the paradoxical question of how one would go about testing the handling of an unknown error.

Having a test case for the handling of an unknown error would make it a known error and would trigger adding appropriate handling, which in turn will invalidate the test case.

I don't see a paradox here. You don't simply remove a test because someone added error handling logic. If you stumble across an error condition that was not previously encountered, you need to have a conversation with developers and management about how this situation should be handled. Sure, you discovered it. Now you wrote a test. Record the expected result, which is the application throws an error, crashes, etc. If this is unacceptable, then requirements need to be defined to handle this.

After an unknown error condition is handled, your old test is likely outdated. The test should be updated to assert the new, expected error handling behavior. Like you said, what was unknown has now become known.

This pattern of testing and discovery is known as black-box testing. It really is a form of experimentation where the tester makes no assumptions about how the system is supposed to work, or how people are supposed to use it. Black-box testing can uncover these "unknown errors". Write a test. Define how the application should behave. Supplement with white-box testing once you understand the internals that allow this error to surface.

There is no paradox. What you are describing is a name for something. The important thing is you communicate what happened, how you encountered it, and have a conversation with the relevant development and management people to determine how this should be handled.

Be aware that their answer could legitimately be "we won't handle it."