0

Using a ton of libraries, IO, ... you cannot handle all imaginary errors. You will need some form of "catch uncaught exception" handling to still have a control flow for such a case. But if you are aware of such an error, you usually can and want to add a proper handling.

That brings up the paradoxical question of how one would go about testing the handling of an unknown error.

Having a test case for the handling of an unknown error would make it a known error and would trigger adding appropriate handling, which in turn will invalidate the test case.

How should we test the handling of an unknown error?

One thought process was to specifical implement a method which can be triggered by some internally known condition which will raise an "unknown exception". But this feels wrong on so many levels.

Any other ideas?

13
  • 1
    If such handling is done by catching an exception, simply catch a top-level exception and in the test suite force the "throwing" method to throw this top-level exception, to ensure you actually have a catch block for it. I do not think anything else can be done about this.
    – Andy
    Nov 9, 2022 at 9:29
  • 1
    What kind of E2E test is this? Client application request against an API, which handles the exception? I.e. the tested client and server are separate apps?
    – Andy
    Nov 9, 2022 at 9:42
  • 4
    handling an unknown error would make it a known error by this rationale, your question is not answerable.
    – Laiv
    Nov 9, 2022 at 10:49
  • 1
    It's unclear what actual problem you are trying to solve here. Why do you think end to end testing of unknown errors is important to your system? Nov 9, 2022 at 10:54
  • 1
    I feel like there is a specific testing strategy that focuses on simulating error conditions that would be useful in your case, but I cannot for the life of me remember what it is called. Years ago Netflix released a suite of tools used to simulate network issues in their micro services. Not really sure if that pertains to you, though. Nov 9, 2022 at 11:08

1 Answer 1

1

That brings up the paradoxical question of how one would go about testing the handling of an unknown error.

Having a test case for the handling of an unknown error would make it a known error and would trigger adding appropriate handling, which in turn will invalidate the test case.

I don't see a paradox here. You don't simply remove a test because someone added error handling logic. If you stumble across an error condition that was not previously encountered, you need to have a conversation with developers and management about how this situation should be handled. Sure, you discovered it. Now you wrote a test. Record the expected result, which is the application throws an error, crashes, etc. If this is unacceptable, then requirements need to be defined to handle this.

After an unknown error condition is handled, your old test is likely outdated. The test should be updated to assert the new, expected error handling behavior. Like you said, what was unknown has now become known.

This pattern of testing and discovery is known as black-box testing. It really is a form of experimentation where the tester makes no assumptions about how the system is supposed to work, or how people are supposed to use it. Black-box testing can uncover these "unknown errors". Write a test. Define how the application should behave. Supplement with white-box testing once you understand the internals that allow this error to surface.

There is no paradox. What you are describing is a name for something. The important thing is you communicate what happened, how you encountered it, and have a conversation with the relevant development and management people to determine how this should be handled.

Be aware that their answer could legitimately be "we won't handle it."

3
  • 1
    Let's say we cannot find any new unknown errors (because we fixed them all). Then how are we going to test the "unknown error handling"?
    – Spenhouet
    Nov 9, 2022 at 20:31
  • @Spenhouet: by utilizing black-box testing. You basically guess. You try things out. You experiment on your system. You can also utilize white-box testing, which affords you some knowledge of internals so you can simulate things like, what if a database operation succeeds but a subsequent file operation fails? You will need to research testing strategies, with a focus on black and white-box testing. Nov 9, 2022 at 21:02
  • 1
    That’s a case for mocking. With mocking you can get any errors. However, you can’t have case specific handling or fix the problem. You can just put everything that is unexpected into the category “unexpected error”.
    – gnasher729
    Apr 10, 2023 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.