What to do when you have exhausted all avenues to fix a bug

Question

I am a Junior Programmer(4 months career experience so far) working on a Cross Platform Mobile Application (1 person team - so its just myself).

I have a bug in this program/app which is pretty large (30 different header files, each with their own cpp file too). I have been trying to track down exactly what is going on with the bug & also to fix it (even tried to use some hacks to just get it working) but of about a dozen or more solutions(ideas I have of whats causing the problem) I have come up with nothing has led me to track exactly what the bug is or fixed the bug.

Do you have any advice for a junior programmer of some broad techniques(go for a run, print all my code onto paper & go through it with a pen, etc.) I could use to assist me with this bug?

To give a little more context for my bug; it involves the cross platform API Mosync, when I perform a specific sequence of actions, the current screen does not redraw (& it appears) that the previously displayed screen is still receiving the pointer/key press events & not the current screen.

Specific sequence:
- Menu Screen Displayed - click "Show prev orders button"
- Prev Orders Screen Displayed - click "Load file" then click menu button & open Delivery Screen
- Delivery Screen Displayed - click menu button & open Purchase Screen
- Purchase Screen Displayed - Error here, input to this screen is not displayed/reacted to, ListViews dont scroll, buttons dont react to clicks, ListView cells dont respond to clicks

---

I will take the advice on board, the bug is reproducable 100% following the same steps each time, although it still very difficult to figure out how pointer events are being transmitted & to what screen due to fact thats a part of the API I cant reach(or dont know how to).

Also I would love to have a different pair of eyes go over my work & point out the bug, but as I said I am a team of 1, my boss directs me, he owns the company & has the ideas for an app but does not know c++ or any recent languages either (cobal? I think is all). Any advice on how to get a second pair of eyes without breaching/showing off the company's intellectual code/property?

...and no leaving this paid internship is not an option, the contract says if I leave before 6mnths of a 12mnth contract I maybe liable to pay 30% of my yearly salary

Answer 1

If you can reproduce the issue 100% of the time, set a break point on the last step (as early as possible). If you walk through the entire call stack, I'm pretty sure you're going to come up to some unexpected values somewhere, or something that should be called but isn't.

Edit:

And if you're sitting at your wit's end trying to fix the bug and posting here hoping that you'll get some shining light advice, walk away. Go clear your head and come back later (preferably tomorrow or after the weekend). There have been many a time that I've spent an entire day searching for a solution to a particular issue just to walk away, come back the next day with a clear head and find it within ten minutes.

Answer 2

Debugging is more about isolating and understanding exactly what the issue is (compared to applying a fix)

One thing to be careful about when debugging is if you start to see that you are jumping after different theories as this often winds of actually taken longer and doesn't systematically eliminate possible issues.

Usually the best way to debug these types of situations is the boring systematic approach by breaking your system down into little pieces and getting each pieces to work in isolation and keep adding in each element of complexity one by one until it breaks. Then you have isolated the exact issue. This way may seem a little tedious and some more upfront work but it removes variables and keep your brain sane while trying to debug a complex piece of software.

Answer 3

These are just some things I have done in the past, obviously they won't all work in every situation:

1. Realize that it's just code, and somewhere in there is a bug (it's not just black magic) that you CAN fix.
2. Take a break.
3. Step through the code very slowly, analyzing each step and making sure you understand it and what it is doing, not glossing over anything.
4. Get a second pair of eyes to look at the problem.
5. Go to sleep and forget about it until tomorrow (clear your head), come with a fresh perspective).
6. Print out your code, and analyze each line, making notes in the margins, understanding every implication of every line
7. If it's not a critical bug, but is causing errors that the user doesn't need to know about, I have (ashamedly, but honestly) trapped the bug, and swallowed it! If it's not dangerous, and you can't find the cause, sometimes you just trap for it and don't let the user know anything happened. It's all about ROI for the client, and sometimes it's not worth it.
8. Tell the bug verbally that you are going to hunt it down and kill it. Sometimes it will run away. :-)

Answer 4

I usually have this approach when solving bugs.

1. Create a nice step by step to reproduce the bug
2. Simplify the step by step
3. Where in the code does the bug occur? Like what functions is involved?
4. What path does the code choose when the bug occur, the callchain.
5. Focus in on the location, when is it ok when is it not. Then repeat this a lot until you found exactly the spot where the error occur.
6. Why does this happen?

At this point it is usually kind of clear what has happened since I learn so much in the process of focus in on the problem so I know what to do. Or I have a quite focused question that I can ask in a forum.

Then I try to fix the problem, and use the step by step you created in step one to verify if the bug is fixed.

Answer 5

All previous advice is excellent, and much of it is aimed at verifying assumptions about the bug/error and then following a debugging process to locate the error (sometimes by examining the environment around the bug and sometimes directly in the code).

This approach will not always work, regardless of depend on your seniority or expertise. Sometimes you just need another set of eyes on the problem. Find someone to review the problem or debugging session with you - often just talking through the code will lead you to the error.

Answer 6

As others said 1) be able to reliably reproduce it, and 2) step forward in a debugger up to the point where it happens.

If I cannot do that, for whatever reason, I have two other methods that both require having a different version of the code that does not exhibit the bug.

1. Run both versions of the code side-by-side under debuggers. Step them along until the bad one does something different from the good one.

2. Alternate running the good and bad versions of the code. Have a diff or some other list of the differences between the versions. Then incrementally change the code of either version to make it more closely match the other. If the bad one becomes good, or the good one becomes bad, I back off the change and make a smaller change. In this way I home in on the bug. I think of it as "getting on both sides of the problem and working toward the center". This method does not require a debugger.

If the problem is hard to reproduce, then I need as much information as I can get, such as a stack dump, when it does happen. So I make sure I can get those diagnostics, wait for the problem to occur, and hope I got enough information to find it.

Answer 7

If you were assigned to do the work on hand as a junior programmer, there is at least one person who believed that you were capable of handling it all by yourself.

Then, before asking for help from your superiors, write down on a scrap paper, the list of steps/methods you took in tracing the bug, how far you went on with it, why you gave up each method, and what have you learned in each attempt. Also, summarize what you have learned about the project so far.

Chances are, when you finish writing this down, what can be done should become blindingly obvious. If it does, you simply have to follow what revealed itself in order to reproduce the bug, and, try fixing. If it does not, you have a grounding on which you can talk with your superiors. If you ask for their help without showing what you have done, they might get a negative impression on you.

But, if you clear up your head, come back after the weekend, you might be able to solve it in no time, without anyone's help. It happens, all the time.

Answer 8

We need to know how hard it is to reproduce, as the method is quite different. For a reliably reproduced defect, automate causing the defect. Use debuggers and debug traces (traces have least impact on race-condition type defects). Get methodical. One step at a time, each step provides more information, even it is confirming what you already know. If you get a surprise result, stop, understand it 100% before moving on. It is painfully slow, but always gets you to the end result if you give it enough time.

If you cannot reporduce it, then you have a problem, how do you confirm you have fixed it. Put in debug code and leave it there. Eventually, ask yourself, is "Closed : DNR" is a valid option? (Did/Could not reporduce). In business, eventually it's a cost/benefit decision.

Don't assume your libraries are correct, confirm they are.

Take a break, be pragmatic about the cost vs need to fix, and above all, ask someone else to sit down beside you and help.

Answer 9

Lots of good answers here. A few other tips:

UIs seldom live in isolation. Build up a test program with the minimal set of features required to reproduce the bug. If the UI is well-designed, you should be able to decouple the UI components which are failing, and run them in isolation in a test program. Can you still reproduce the problem? If so, the problem is likely in your UI structure or framework. Check your UI structure - especially watch out for invisible elements. Try to learn exactly what happens when you click on that ListView and it doesn't respond - what event handlers are invoked? Bear in mind, there may exist bugs in the UI framework itself - don't jump to that conclusion, but don't rule it out outright. A quick test is to upgrade your version of Mosync and check if the symptoms abide.

Failing that: What is left in your test program? Understand all the components of what remains, particularly any running threads. Something doing database maintenance in the background? A file spooler of some kind? NSA user behavior monitoring code? Is the UI working with some of these components (possibly behind the scenes)? What background operations does the UI depend on?

While you're reading the code - which you should be spending considerable time doing, given the difficulty in the bug - watch out for some bad practices that could be obscuring your bug. Specifically, do you see any of this?

try {
    SaveTheWorld();
} catch (std::exception& ex) { /* oh it didn't work, let's just ignore it */ }

That's incredibly poor practice and as such, is fairly commonplace (hey look it didn't crash!). Make sure you upgrade any code that's doing that to at least log it - preferably remove the false exception handling entirely. (A rule of thumb is that if you don't know what the exception is, you're not prepared to handle it.) If it's interacting with C-style APIs, watch for dropped error-code return values, and be sure that you're checking error status information from whatever tools you're interacting with.

Seeing as how your test program is now properly handling failures, and you've read the log you've thus produced, but still nothing highlights the bug, look for interfaces you can probe. Is there a network transaction that should be happening under the covers? If so, hit it with Wireshark. Database transaction? Try some query logging, or checking the database server status. Filesystem or network shares being hit? Check the intermediate files, or use a debugger to trace I/O. Hardware I/O? Monitor and probe. Be empirical. The UI could well be hung up on some background operation you haven't anticipated.

Lastly: Don't panic. Keep cool, and keep track of what you've tried. If you still can't find it, it'll have to become a "known issue" to be tracked down on a rainy day. You'll want lots of material to justify that decision if it has to go that way.

Answer 10

In the scheme of things, reproducible bugs are (relatively) easy! Why? Because you can always hack down the code to the bare minimum until the bug disappears, and then work back to figure out what code causes it. So that's one method. It's reproducible, you have the critter there under your control. You can poke it, and experiment with it. You can even dissect it if you want to.

Your first objective is to understand why the bug is happening in your code. Don't try to fix it initially. Just try to understand it. If you try to fix it without understanding it you'll be hacking around and will likely introduce technical debt, even if you solve it.

Step through the behavior of the app, line by line. Watch the variable values. Watch the flow of control. Where does the behavior first deviate from what your understanding tells you that it should be? Do you understand how the operating system sends events to your app? If you are hampered by "black box" problem, can you get the source for compiled libraries/frameworks, enabling you to step through at a deeper level if you have to?

Do you have a commit in your version control system which does not produce this bug? (You are using version control aren't you?) If you do have such a commit, then you can do a binary search on the history to find out exactly where the bug was introduced.

Your objectives should be to (1) understand - determine the cause and to that end, try to (2) examine, understand the app behavior in detail (3) isolate the issue by making it go away and then examining and understanding the delta that enabled you to do that

But definitely don't sit there for weeks if you are really stuck. You have to tell someone in your organization too. Solicit help where you can and past a certain point, it is certainly incumbent on you to tell management that you feel you've hit a barrier to progress. But you will likely be able to resolve this if you hit it from a number of different angles, all focused on learning and understanding.