I'm not even sure how to define this difficulty. It reminds me of the test a couple of prospective employees did on me before I got a job. They would pick an object in the room and then I'd be allowed to ask questions to help myself determine what that object is (much like 20 questions). I was ridiculously good at this (no, I never got high points for humility), so I'd assumed that I'd be really good at troubleshooting bugs...

But here's the thing I figured out recently. I'm really good in that situation because it's really easy to see everything that's in the room, thus I can approach my problem with some concept of it's component parts. In essence I "know what I don't know". But with programming I run into a lot of situations where the problem is a complete unknown to me. I know it's broken, but I have no concept of how it might be broken. I've followed all the instructions, I know the technology fairly well...

If I'm honest, I feel like I'm just having a hard time imagining things that could be wrong so I can test them and, hopefully, find a solution.

How do I go about developing that skill? What do I need to do to help my, apparently, limited imagination come up with ways that my project could possibly be broken? Are there exercises (puzzles perhaps?) that can make me better at this? I'm aware that probably the biggest cure is just experience...but I'm hoping to help accelerate the process if I can. Staring at my computer screen blankly for a few hours at a time is not even kind of fun...

  • 3
    imagine how you think it might work, and work backwards from outputs to inputs to find paths to investigate Apr 23 '14 at 20:48
  • 1
    I'll toss a link out there - How to be a Programmer - the first skill listed is "Learn to Debug".
    – user40980
    Apr 23 '14 at 22:47
  • 1
    I wanted to toss out something regarding the "out of the box" thinking. With regards to bugs, I often think that the first thing to do is to simply list all the systems interacting, then assume any part of it could be at fault until proven otherwise. Then your job becomes easier: assume that component is failing, and find a way that it could even if at first it seems illogical ("output is being corrupted",etc). Then prove your component isn't failing, starting with the most immediate interactions. After the fact it can seem like imagination but often it's just starting with a pessimistic view.
    – J Trana
    Apr 24 '14 at 4:11
  • Write a printf or println or whatever you use under every line of code to be 100% sure everything works how you want it to work haha. Then run your console application with App > out.txt then comes the hard part viewing the huge file.. sometimes my log files are over a few million lines and it may take some time haha. Of course the right way would be to use a debugger and breakpoints but sometimes it's not possible to do that.
    – SSpoke
    Apr 24 '14 at 5:58
  • 1
    Read Pirsig's Zen And The Art Of Motorcycle Maintenance amazon.com/Zen-Art-Motorcycle-Maintenance-Inquiry/dp/0060589469 Apr 24 '14 at 6:46

Every defect in software is ultimately due to a discrepancy between assumptions and reality. Insidious bugs are simply discrepancies between particularly deep-seated assumptions and reality. The ability to diagnose bugs depends on your capability to question your own assumptions, and that does indeed require an awareness that you don't know certain things, you just assumed them and used to be right until now.

Obviously the tools of the trade, log files, debuggers etc. are helpful in uncovering such assumptions and re-aligning your world model with the actual system. But until you are ready to question the crucial assumption, e.g. "It can't be bad input because we have comprehensive input checking", you will spend your entire time checking the wrong parts of the system, or simply not knowing where else to look in the first place.

  • 3
    I hate to say it Killian, but I think you've hit the nail on the head. I've gotten very proud of my "knowledge" of the systems I've acquired over the time I've been here and I think I'm mentally resistant to the idea of being wrong. As I may have mentioned, I never scored high on humility. Following your advice to challenge my own assumptions actually allowed me to make some solid progress on a couple issues I was facing in my own code. So, thanks, I'll bear this in mind going forward.
    – Jay Carr
    Apr 24 '14 at 3:18
  • 2
    @JayCarr: "mentally resistant to the idea of being wrong" - What if you try to see mistakes as a source of learning rather than a fault? There's nothing wrong with being wrong, as long as you don't stop there.
    – JensG
    Apr 24 '14 at 9:20

What do I need to do to help my, apparently, limited imagination come up with ways that my project could possibly be broken?

In most cases, I'd say absolutely nothing. You shouldn't be trying to dream up things that might be causing the program to break. You should be systematically determining what is causing it to break.

You should be going into the debugging process with the following pieces of information:

  • the steps that were taken and the values that were entered to produce the bug;
  • what the program should be doing when given those steps and inputs;
  • what the program is doing when given those steps and inputs.

If there's an error message, get all the information you can about it. If the error message itself isn't clear, and you don't know what it means in practice (some error messages aren't always particularly useful), then use Google, or StackOverflow, or any other online resource to find information about it.

If there isn't an error message displayed on the front end, check any logs the application writes to for error messages during the time period you reproduced the bug. The code may have executed to completion, but encountered an exception that's being handled along the way which is throwing off the end result and producing an entry in the logs. Look for those, do the same above and identify exactly what they mean.

If there are stacktraces provided with exceptions thrown by your code (and there should be), look at the lines of code mentioned. The line itself may not be the one that's actually producing the issue. If you get a NullPointerException in a piece of Java code, the stacktrace will tell you where you tried to use something that was null when you expected it not to be. That doesn't exactly point you to the line causing the problem, but it generally does tell you what variable doesn't have the value you expect it to, so you can look at any references/assignments to that variable to determine that the value isn't being set or that the value is being set incorrectly.

If none of that has helped, fire up your debugger. If you've narrowed it down to a section of the code that you know is causing the issue - but you don't know exactly which line(s) - then step through that. If not, just step through the entire thing. This is where you need to know exactly what the program should be doing with given inputs, because you need to look at each value after each line and determine exactly where it's deviating from what you're expecting it to do.

Still don't have a clue what the problem is? Ask somebody for help. A co-worker, a friend, an online community. Show them all that work you just did. Show them the error messages, the stacktraces, explain what the program does in general terms (if they don't already know), what it should be doing in this particular instance (e.g. returning the value 4), what it's actually doing (e.g. returning the value 5). If you've narrowed it down to a few lines of code in the debugger, say "I know the issue is caused by these lines in the code, it's setting the value to X when it should be Y, but I can't see why that's happening".

Spending a few hours staring blankly at your screen definitely isn't fun, but there's no reason you should be doing that. If there's a problem with your code then you need to be reading or stepping through the code.

  • May have been a bit quick to judge this answer, was a bit frustrated when I read it. Sound advice. Killians comments just spoke more to the heart of my problem I think. That's the only reason this isn't the selected answer instead.
    – Jay Carr
    Apr 25 '14 at 13:27

To some extent, it is like investigating an criminal case, or a mind-boggling puzzle.

First, you got the victim. After digging a while into the case, you identified a few suspects and you also developed a work hypothesis how exactly the victim might got murdered. You continue investigating, looking out for more helpful information, getting you nearer and nearer to the real source of the problem.

It happens that from time to time you walk into a dead end (pun intended). That's part of it, and there's nothing wrong with it, as long as you manage to bring yourself back on track as quickly as possible. The key is, to always think about what piece of information you need next, that either supplies your hypothesis (and provides you with further information), or proves it wrong. Then find a way to get that information in an efficient manner, make your conclusions and move on, until you finally are able to convict the guilty.

And sometimes you realize, that all the facts and indications necessary to spot the culprit were already waiting in front of you half an hour ago. Albeit annoying, that's among the most interesting parts, because if you do a critical review of your actions and mistakes, you may be able to learn and get better. Ask yourself questions like:

  • How could I have avoided this waste of time?
  • What did I overlook in the first place, and why?
  • What unvalidated and/or wrong assumptions did I rely on?

This will train your abilities. It will also develop your gut instinct, so over time you learn to automatically notice all those minorish things that are too easily overlooked, leading you quicker to the right answer. At the end, it's all about deliberate practice.

Last not least always remember what Sherlock Holmes taught us: When you have eliminated the impossible, whatever remains, however improbable, must be the truth.


What do I need to do to help my, apparently, limited imagination come up with ways that my project could possibly be broken?

Let history be your guide. If your project is well managed then you should have a database of every bug that has ever been fixed in the product along with an analysis of how the bug was found, how it was reproduced, how it was analyzed, and how it was fixed. That is not a write-only database. Read the database, and very soon bug taxonomies will begin to occur to you.

That will give you a good overview of the sorts of things that go wrong in your product. If you are interested more generally in what goes wrong in all kinds of software, particularly with an emphasis on security-impacting defects, I suggest that you read the CWE list: http://cwe.mitre.org/data/index.html


So rather than trying to reproduce and fix a specific defect I believe you are asking about thinking up new tests that you could use to investigate the product to see if the product works under those circumstances for example: what happens if I enter special characters into our sign up page's password filed, or what happens if I forcefully close the program while it is writing to the database. These cases are indeed hard to think up.

Software development over the past 10 years (Agile / XP / TDD etc.) has come to value meeting the explicit requirements only and then declaring the feature finished, and not finding every possible way to break something (there are possible exceptions, if you are working for NASA or doing white hat security but even then there is a case to be made for calling such things out in the user story's acceptance criteria).

So if your features explicitly list as acceptance criteria what the systems needs do such as how to handle input, its performance characteristics, user workflow actions, etc. then you have a complete list of what the tests need to be checking for. Testing should be done to validate that requirements have been met, and the best way to do that is to explicitly list all of your requirements. Take a look at the Agile Test Quadrants.

Some people advocate for these tests to be the explicit declaration of the requirements, that need to be written before the code i.e. Test First ( or Test Driven Development).

However I appreciate that you do not sound like you are contemplating a new project where you can set your own development best practices before you start and are instead coming in after the software has been written and are being asked to 'test it'. This is indeed more challenging but the same principles apply, you can only test it once you know what it was supposed to do. If there is not a comprehensive list of requirements that were met by the development team for you to work from then asking questions is the best way forward. Depending on your team this may need to be done delicately as people who did not explicitly list their requirements before building a software system do not like to be reminded of what they missed but it is essential to performing this task well.

Once you find a requirement - it must be robust / it must be secure, then try to dig deeper and find out how secure it must be, or how much failure is acceptable, there is always a limit, not many people have an NSA proof level of security as a requirement for their application or would want to pay for that. The more you dig the clearer it should be as to what types of security attacks you need to be defending against or easy of use you are aiming for. Some domain knowledge is then useful, in security, in design, in performance etc. which is where you ask even more questions from the experts you can find in your team, or here on SE, or google / books.

  • Not quite the question I was looking to have answered, but excellent commentary none the less. I'm voting you up, this is very useful commentary.
    – Jay Carr
    Apr 24 '14 at 3:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.