How to convince team members of the existence of a "mandelbug"

Question

We are developing an application; it includes a library developed by another coder, this library communicates with server via multiple network connections, and this involves multiple threads working together. The server side code is quite complicated, and we don't have access to the source code.

Recently I've discovered a mandelbug making application crash sometimes. I could reproduce it once and got a stack trace, so I opened a bug report. The bug itself is easy to fix (uncaught web exception in one of background threads, that makes CLR terminate the program).

The problem is that developer is refusing to fix the bug, because "he is not convinced it exists". Unfortunately for me the boss is siding with him and says this bug cannot be fixed unless I make a "solid test case" to prove existence of the bug, and to make unit test verifying that it's gone. What is basically impossible due to a nature of the bug.

Any advice?

Answer 1

If possible, may be spend some time to check if this defect can be reproduced by putting some sleep or block in your application code. But do not spend too much time. As this issue is due to multi-theading (and also as you observed), it's occurrence will be rare.

My advice is not to sweat over this too much. Continue your work. Whenever you come across this crash, update your bug report with the stack trace, saying that this is repeat occurrence, and changing the owner to library developer. Let the management/lead decide whether to fix or not depending on it's frequency.

Also try to understand the developer's mentality. You said "uncaught web exception". The developer at this stage may not be entirely sure what will be other effects of catching this. So he/she may be is reluctant to touch the code.

Answer 2

So, from your more or less clarifying comments, I got it this way:

You are sure there is only a simple additional exception handling missing, and you know already which code line in the lib is problematic and how the lib could be fixed.

Why then don't you just add the few missing lines of code to the lib by yourself, ask the team kindly to test the lib with that changes? Make sure it is a low-risk change, easy to understand by the dev who is responsible for the lib. The worst thing that could happen is that someone has to revert that change in your VCS if your fix causes some new unexpected behaviour.

Most people are easier to convince when the work is already done. Also, they react better on "here is an improved solution", opposed to "this code is wrong, fix it somehow".

EDIT: when the dev still refuses to add that change, best option is trying to make the problematic code work inside an isolated test harness where you simulate the network error. Working effectively with legacy code describes a lot of techniques on how to tackle such kind of problems. For example, you could create a test version of the library, including only the problematic modules and functions, and create a "mock environment" around it where you can simulate the "network exception" under controlled conditions. That may seem to be too much effort at first glance, but once you have such an environment, you can add a lot of additional tests to it (and I guess that will make sense, since when the author of the lib refuses to add missing exception handling in one place, one should expect more bugs of that kind).

Answer 3

For a bug such as this, automated fuzz testing (also called random testing) might be helpful in trying to reproduce it. This automates the process of finding the bug by randomizing a fixed set of parameters (or inputs) into the thing you're testing. Each test run, parameters are recorded to a log file, including time stamps, etc. so that when the crash occurs you can (theoretically) just replay back the test, using the same parameters, to reproduce it.

Since its automated, the test process can run many tests in a short period of time. Often it can be left to run overnight, and in the morning you can check a log file to see if it reproduced the crash.

Answer 4

The Devil's Advocate suggests another path.

The other developer has stated, flatly, that there is no bug there.

Can you come up with a way to stress the hell out of his allegedly-nonexistent bug, and cause it to crash a lot more frequently?

Answer 5

The stack trace is clear evidence the bug exists, or at least did exist in a certain build. What you don't have is evidence the bug was fixed. They are foolish to ignore it. I've had "impossible to reproduce" bugs after hundreds of thousands of automated tries on multiple systems that triggered every single time on a customer's system.

I get a couple bugs like that per year, most without even the benefit of a stack trace. In nearly every case, even though I couldn't reproduce it beforehand, I could pretty easily make an automated test for it once it was fixed.

For example, a few months ago I fixed a bug that only occurred when the user typed faster than 96 words per minute. Before I fixed it, all I knew was that the bug happened "sometimes." It would never occur to me to write a unit test for fast typing. However, after I knew the root cause, making a test for it was trivial.

Even in those rare cases where a bug can't be reproduced even after being fixed, you can close it by code inspection.