Obviously, the easiest way to solve a bug is to be able to reproduce it in-house. However, sometimes that is not practical. For starters, users are often not very good at providing you with useful information.

Customer Service: "what seems to be the issue?"
User: "It crashed!"

To further compound that, sometimes the bug only occurs under certain environmentally conditions that can not be adequately replicated in-house. With that in mind, it is important to build some sort of diagnostic framework into your product.

What types of built-in diagnostic tools have you used or seen used?

Logging seems to be the predominate method, which makes sense. We have a fairly sophisticated logging frame work in place with different levels of verbosity and the ability to filter on specific modules (actually we can filter down to the granularity of a single file). Error logs are placed strategically to manufacture a pretty good representation of a stack trace when an error occurs. We don't have the luxury of 10 million terabytes of disk space since I work on embedded platforms, so we have two ways of getting them off the system: a serial port and a syslog server.

However, an issue we run into sometimes is actually getting the user to turn the logs on. Our current framework often requires some user interaction.


We have two tools to handle that where I work. The first is an exception reporting tool. You add it to the project file and make sure the linker generates a map file, and when an unhandled exception gets raised it will gather information, write out a crash file, and email it to us.

The second is logging. By making a log of significant execution points, we have a roadmap to what the EXE was doing just before an error occurred, which can really help track down problems.

Try adding these two features to your project and see if they don't help you track down errors at the client's site more easily.

  • Wow..that first one is sexy! – Pemdas Jan 7 '11 at 21:56
  • @Pemdas: Yes, it's very nice, especially because it provides full stack traces. That makes debugging so much easier! I'm not sure what you're coding in or what options are available for it, but we work in Delphi and I can name two different packages that will do that for you: MadExcept and EurekaLog. – Mason Wheeler Jan 7 '11 at 21:59

There are various answers depending on the level of control you have over the target platform and the trust between you and the client.

If you are tightly bound, you may be able to do actual remote debugging. This is unlikely to occur.

Your options quickly devolve to log files and crash-dumps.

You should likely have some kind of logging framework already in place in your code. If you do not, pick one appropriate to your language and environment. Make it usefully toggle-able between various levels of logging. Pay attention to what you log - remember you will be psychic debugging from the very strings you emit in the log.

Alternately, you can use crash-dumps - like core dumps on unix or minidump on windows. These store the internal state of your program at a point in time. You can then load that into a debugger locally to see the state of your system before it died.

Hope this gives you some general hints as to getting started.


On one of my co-op's I worked on an embedded platform developed in c++. They implement some method of specifying what class data/variables would interesting to look at if the system crashed. I am not exactly sure how they did it, but it would essentially dump all the marked data to flash when the system crashed, which could then be extracted via a serial port and sent to support.


Have it dump everything that it can into something that can be sent (optionally) to you the next time it runs. That is the best possible way that I can describe to get insights into what happened so that you spend the least amount of time possible on a customer's computer.

Just like any science, simply observing things often results in different than typical behavior than normal. You really want the program to tell you why it died.

I work mostly with UNIX like operating systems, so my examples might not be a quick fix, but hopefully they illustrate the thinking:

  • Handle any fatal signal or error that you can handle and log it when appropriate. You need to know why a program (or its kernel) decided that there was just no more point in living.
  • Have 'odd' but non-fatal things in your program notify the user that additional logging will take place, and that things don't seem to be going as planned. This helps save user confidence while reducing on brain picking from vendors.
  • When asking users if they want to send error reports, specifically say "This may result in a fix for this problem being issued shortly", and make sure that fixes do come of it most of the time. Don't word it like "This helps us improve" , as that is problem agnostic and not what an irate user wants to see.
  • When possible, use checkpointing. This lets you replay and recreate the issue until it is solved. In the day and age of virtualization, this is becoming a broader possibility.
  • Realize that clients tend to 'clean up' a bit prior to offering their PC to someone else. You might not be able to reproduce the issue, even on their computer. Just wonder what isn't there that was there before.

You must be transparent when gathering data, but you'd be amazed what people will give you if you can convince them that doing so is for their immediate benefit.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.