Unstable output C++: running the same thing twice gives different output

Question

So, the problem:

When I run the same C++ code in Visual Studio, with the same input and parameters, I get either the correct output, or an output that is completely messed up (99% of values go to zero).

Now, the code is some very complex, hard to read legacy code that is used for over 20 years and developed by many different people, containing 100k lines of code. I also made some minor changes for one project, that probably caused this unstable behaviour. Not even sure that it's induced by me, but I guess it must be the case. I was thinking of memory issues, maybe because I assigned more elements to some variable that has not that many elements allocated in memory. This normally gives an error, but... . Or might be something else low-level/mid-level, but not sure what.

The problem here is, I have 3 options:

1. roll back to a stable version and introduce my changes one by one and test again
2. find the root cause in some systematic manner
3. Guess and be lucky that I find an error that I'm not even sure what is might look like

The problem, respectively:

1. Sometimes the program works well for 20 runs at a time. So, it's gonna take forever to test whilst adding each change one by one and testing each new change... .
2. I have no idea how to systematically tackle these kind of issues (I'm not really a programmer, I just have a lot of experience with high level programming like MATLAB, R etc.)
3. Luck might not be on my side.

Anyone has any idea how to help with option 2? Like, what kind of issues could possible cause behaviour like that? How can I tackle this, e.g. if it does not work during a run, does it make sense to dig into the code (even thought next run it might work perfectly), should I look at warnings, errors in the output window, etc. ?

*I also have this message in Output-->General:

Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.*

I know you don't have all info, but I cannot share my 100k lines of code program here, not,if I would, would you be able to understand much of it without hours of digging. So it's a general question.

Update: when compiling/building the .exe and running, the wrong output does not seem to happen.

Cheers, thanks!

UPDATE: SOLVED, and thanks for the advice!

So, how did I solve it:

• I automated generating lot's of output
• I rolled back until I could find a version that was stable

I already had some things in mind that could cause the error, and this matched with the changes made since the latest stable version. I then deactivated some usual suspects, and the output became stable again. The line I deactivated corrected some wrong input, but only for the product I was working on. I still didn't know exactly why, because the part of the code I deactivated was not directly related to the actual line of error. But somehow it "solved it" (as in, 2 bugs (wrong input for the new product, and one out of bounds memory bug I had created through recent code changes) did actually compensate for each other). But I was close and then I found that from the product I was working on j.m_res_inv[i] went out of bounds, so I created an if to subtract one from the maximum index.

 if (b->aidnprdl == FIB_VARIABLE || b->aidnprdl == AGI_VARIABLE)
 {
     for (int i = 0; i < b->nbre_sc - 1; i++) // don't assign random memory values for a subcontract that does not exist yet ;) and make b->nbre_sc equal the number of actual branche21 subcontracts in the input data
     {
         b->m_res_inv[i] = j.m_res_inv[i];
         b->res_inv_br21 += j.m_res_inv[i];
     }
 }
 else
 {
     for (int i = 0; i < b->nbre_sc; i++) // original code: b->nbre_sc equals number of branche21 subcontracts
     {
         b->m_res_inv[i] = j.m_res_inv[i];
         b->res_inv_br21 += j.m_res_inv[i];
     }
 }

What I still don't understand:

• Why does C/C++ not crash when this happens? The random memory location is probably not containing the same variable type (double, int) as needed by j.m_res_inv[i] .
• Even if it works, why does it not put just a random value there?
• Strange thing is, when unstable, every output value (like, an .csv of 2000 rows, 300 columns) would go to zero. This probably depends on particularities of this program but still I would expect one "weird" value not to corrupt each and every computation that in theory, are unrelated to this value (mathematically). Maybe there is some particularity in the program that when somewhere a negative value is encountered it stops computing. Kinda hard to find out was was happening I find.

Anyhow, thanks a lot for all the good advice!

p.s. I marked one answer as definitive, but it's the combination of answers that was useful for me!

Answer 1

Sometimes the program works well for 20 runs at a time. So, it's gonna take forever to test whilst adding each change one by one and testing each new change...

First, you need to work on this. Find a scenario which gives you a much higher probability of crashing the program:

• maybe by finding more critical test data,

• maybe by adding asserts or validations in sections where the wrong behaviour shows up and can be identified programmatically

• as a last resort, if you know one from 20 executions of the program shows a crash, automate the repeated execution / restart of the program up to, lets say, 100 times

Since we do not have any idea what your program does exactly, there is no way to give you more specific advice, but reproduceability is always mandatory to find the root cause of the bug - there is no shortcut around this. Without a reproducible scenario, one will not be able to verify a program change has really fixed the bug sucessfully, regardless which method one uses to identify a possible solution.

• roll back to a stable version and introduce my changes one by one and test again
• find the root cause in some systematic manner

Rolling back to a stable version, then introducing changes one-by-one (and re-run the test scenario from above, of course) is a very systematic manner to find the root cause of a bug. In case all of the changes were added as individual commits in a Git repo, one may be able to speed this process up by utilizing git-bisect. But even if that is not the case, changing one step at a time and then running a test which reliably shows the presence or absence of a certain bug is definitely the most effective way to chase this kind of issue.

Answer 2

Cherish this bug. Do nothing to harm it. There is no reason to suppose it is new. It could well be a bug which has been in the software all along, and which has now been revealed as a result of your changes. Now that the bug is visible, you have a chance of finding the root cause.

1. Roll back to a stable version - Definitely not. The bug may already have been there.

2. Find the bug - yes, this is what you must do.

3. Accidentally make a change which seems to make it go away - no, on no account do this. To take an example: if the crash comes because something is receiving a null pointer when it shouldn't, do not remove the crash by testing for a null value.

The problem you have is with the 19 times out of 20 that the bug does not happen.

First, automate

Arrange a "test harness" which runs the program N times and counts how many times it crashes. For instance, if your approximate figure is right, N=100 should report 5 crashes.

For all your future work, choose a value of N which will reliably give at least one crash.

Next, log

Create a log file. Insert, at the start of one or two likely-looking functions, a call to write a message to the log file, giving file or function name or line number, plus perhaps an indication of an argument or two.

Your test will now yield N log files, some of them associated with a crash and some not.

Check that your N=100 (or whatever) still does bring the expected number of crashes. This is important because many of the possible causes of a crash will be to do with uninitialised memory, and any change at all might accidentally stop the bug from happening.

Beyond and before

Are all N logs identical? If there were no bug, they ought to be.

If they are in fact identical, even as between the crashing and non-crashing runs, then you have not logged enough. Log more, especially close to where you think the crash may be happening.

Remember that storage is cheap and comparison tools are fast. There is no harm in adding and adding log calls until each log file is 100MB in size or even more.

Once you have got near the "sore spot", you will probably find that there are three cases:

1. Runs normally with a normal log output.
2. Produces an abnormal log output but does not crash.
3. Produces an abnormal log output and crashes.

The point here is that 2 and 3 combined will probably be much more frequent than 3 alone. (An example is when a byte is uninitialised and can have any random left-over value, and a value of 0 causes the crash.)

You do have to make a judgement as between "normal" and "abnormal". You might take, as a working hypothesis, that if many logs are identical and only a few are different, the majority are the "normal" ones.

Work backwards

Your strategy now reverses itself. Earlier on, you were working forwards in time, closer to the crash, to find a time when the logs disagreed. Now you go backwards in time, starting from the first moment when the logs disagree and going back earlier and earlier to find where the disagreement started.

Do not correct!

The biggest temptation is this: you find a point where something is obviously wrong (eg. is zero when it shouldn't be) and you modify the program so that it avoids a crash in that case. On no account do this.

You may, if you wish, put some sort of "if zero, then return" self-protection here, but you must turn it off for now. You need to find the cause, not cure the symptom. I usually put the self-protection in but turn it off with an #ifdef. Once you have found the real cause, you can turn this protective code back on. But not until then.

Answer 3

Update: when compiling/building the .exe and running, the wrong output does not seem to happen.

That is a major clue. When you're debugging, you're almost certainly debugging a debug build.

Debug builds link in the debug run-time libraries - one of the features of Window's debug run-time libraries is they overwrite memory when it's freed. So code that works in a release build because the freed memory it accesses still contains the expected data will fail in a debug build because the freed memory was overwritten with garbage.

As mentioned in some comments, you can use Dr Memory or Visual Studio's Address Sanitizer to find memory access errors like that.

I strongly suspect you'll find a very large number of problems. Every one found is a bug that has been latent ever since the code was written. And every single one must be fixed.

Answer 4

There's already a lot of good advice here in answers and comments, a final answer might be a meld of different things. Since you added that the error condition happens only when debugging and not when you build and run, it sounds like brute-force approaches to reliably reproduce the error may not work. I see you're going to try, so good luck, that's valuable experience to gain.

If you come up empty-handed though here's another idea. I'd look at the differences between what happens when you build and debug versus when you build and run. You mentioned the error condition leads to most values getting clobbered down to 0. So as an example of what I'm talking about, the first thing I'd probably do is look at VS's 'security checks', specifically the /sdl- switch (Project Properties | C/C++ | General). One thing it does when on is pre-initialize memory before use. If it's currently off in your project config then I'd see how things behave with it on, and vice-versa. If that doesn't increase reproducibility I'd look at similar differences between run and debug configurations. One difference I'd look at sooner than later is what DimitrijeCiric mentioned--possible timing issues (a debug run may introduce a slowdown that sometimes reveals the timing issue), and his advice sounds solid to me.

Also, I'm not sure that reverting your changes is a bad thing, I think you could treat it as another stimulus in isolation to see how it changes system behavior. Especially if you're nervous about your changes you can revert and see how things look, it won't bite. It doesn't sound like you have enough insight into the bug to have it obscured by more variability at this point.

Congrats by the way. You were a self-proclaimed "not a programmer", but if you weren't one before I'd argue you're discovering the one in you now. This is the typical intro.

Answer 5

One key advice for dealing with such problems has not been mentioned yet: Run the code using the valgrind memory checker if possible!

The point is, the C famliy of languages, along with some other languages, are not memory safe: While they do define what valid memory accesses are, they do not include any checks or safety nets to save the day. They trust you, the programmer. Absolutely. They say: You are the human, you are the master, you are more knowledgeable than I, you tell me what to do and I'll do it. And they follow through with that. You access an array out-of-bounds, they do it. You use a free'd object, they do it. You use a variable before you've assigned a value to it, they do it. You are in power, and it's you that is to blame for anything that goes wrong. And if you do screw up, an immediate crash is actually the best possible outcome that can happen. Much worse are unpredictable garbage results like you experienced, or even just subtly wrong results that can't even tell it's wrong. The upside is that these languages do not waste their time with pointless bounds checking...

Now valgrind is a tool that runs any code on a virtual CPU that does perform all the memory checks that are still possible on an already compiled C/C++/... program, and generates appropriate error messages for them. And it can check a lot. The expectation is, that for any memory corrupting code, valgrind is likely to point you directly to where the first memory corruption is happening, and thus dramatically shorten the time it takes you for moving from random observations to deterministic causes.

However, valgrind is not perfect. It will not find every kind of memory corruption. And conversely, even more anoyingly, it will also find shady code in linked libraries that happens to work fine anyways. For some codes, this may render valgrind pretty much useless. And since it runs the code on a simulated CPU instead of the real one, it incurs a hefty slowdown of something like 50x-ish. Nevertheless, it's the first thing you should think about when you are confronted with memory corruption: Is it possible to run this code under valgrind?

Answer 6

For me, a legacy system is a system that has no tests. So, in the long run, to reduce a change of a bug, implement testing at various level: unit, functional, integration tests.

For a quick fix, there are few options:

• trigger the bug using a memory sanitizer, like valgrind
• go back in commits until the bug is gone, like explained in this answer which explains how to do "git bisect"