Spending too much time debugging

Question

Yesterday, I rolled out a v1.0 release of a Web project I've spent about 6 weeks working on (on and off, that is). I haven't made any exact records of my time, but according to my experiences I would estimate that out of all the time I spent programming, half of it was spent debugging. I estimate that to be about a good 15-20 hours spend debugging, which to me is precious time that could have better been spent writing new code or finishing the project earlier. It also especially doesn't help that I'll be a freshman in college in 5 weeks.

The thing is, I feel bad for spending all that time debugging. All that time spent debugging makes me realize that I made some pretty stupid mistakes while I was developing my project, mistakes that cost me a damn good amount of time to fix.

How can I prevent this from happening in the future? I don't want to spend 50% of my time debugging, I'd rather spend 10% debugging and the rest writing new code. What are some techniques I can try to help me reach this goal?

Answer 1

You're asking for the Holy Grail of software engineering, and no one has "the" answer to this question yet.

What is essential is that you track the types of errors that you're making and then do an analysis of those errors to determine if there is a common trend. Root cause analysis is the formal name for this type of introspection, and there is plenty of material on the web regarding it.

Professionals use a bug tracking system so that they can (1) know what needs to be fixed, but also (2) analyze what had to be fixed after-the-fact. You don't need to be so formal -- just keeping a tally in a notebook may be fine for you.

Design Stage Defects

If you find that most of your errors come from a misunderstanding of the problem statement, or if you keep finding you've chosen the wrong algorithm or path to follow in solving your problems, you have problems in the design stage.

It would behoove you to take more time at the beginning of the project and write out exactly what needs to be done and how it should do it. Review this work carefully and revisit the original problem and determine if you really are tackling it in the right way. An extra hour or three at the start may save you many hours down the road.

Coding Errors

If your design is solid, but you're constantly fighting the language that you're coding with, get yourself some tools which will analyze your code for you and warn you early and often that you're making mistakes.

If you're programming in C, turn on all compiler warnings, use a semantic checker like lint, and use a tool like valgrind to catch common dynamic-memory related issues.

If you're programming Perl, turn on strict and warnings and heed what it says.

No matter which language you're using, there probably exist many tools out there to help catch common mistakes long before you reach the debugging stage.

Integration Stage Defects

As you develop your code following good modularity practices, you have to begin gluing the separate pieces together. For example, different sections of your code may have to do with user input, database interaction, data display, algorithms/logic, and each of these are built relatively independent of one another (that is, you tend to concentrate on the section at hand rather than worrying about integration with everything else).

Here is where test driven development (TDD) comes in very handy. Each module of your code can have tests which verify that they work according to how they were designed. These tests should either be written first or very early in the process so that you can have a set of "helpers" to keep you honest. When you begin making everything work together, and you find that you're having to change how this or that is implemented or interacts with another sub-system, you can fall back on your tests to make sure that what you've done to make it all work together doesn't break the correctness of the code.

And So On...

Pick up some books on software engineering and practical coding techniques, and you'll learn many different ways of making development less chaotic and more reliable. You'll also find that just plain old experience -- earn a degree from the school of hard knocks -- will get you into shape as well.

What almost everything boils down to is that a little time and work upfront pays off in huge dividends later in the development/release process.

The fact that you've noticed these issues so early in your career speaks well for your future, and I wish you the best of luck.

Answer 2

Write Unit Tests

Writing unit tests for your code will force you to think about your architecture and encourage you to write your code in small, carefully controlled, testable pieces. This will greatly reduce your debugging effort, and the small amount of debugging you do perform will be confined to small, tightly focused pieces of code.

In addition, the tests you write will "cover" your code; you will be able to tell when a change you make to the code breaks something, because one or more of your existing tests will fail. This reduces the overall complexity of your debugging effort and increases your confidence that the code works.

The catch, of course, is that your time spent debugging is now spent writing tests. But you only have to write them once, and they can be executed as many times as necessary after writing them.

Answer 3

50% for debugging (in a broad sense) isn't all that bad. People typically spend much more time designing, testing, fixing bugs, refactoring, and writing unit tests than you do writing the actual code. It's part of the job.

And to be honest, it's much worse in maintenance programming - quite often, I'd spend an hour figuring out what exactly goes wrong, then five minutes writing the code to fix it, and then half an hour testing the whole thing. That's just over 5% coding vs. almost 95% non-coding.

There are a few things you can do to reduce debugging time though:

• Write debuggable code. This means: proper error handling (with some thought put into it), structuring your code to make it easy to follow, using asserts, traces, and whatever else can make the debugger's life easier. Avoid complicated lines; a line that does more than one thing should be split up so that you step through them individually.
• Write testable code. Split your code into simple functions (or whatever else your language of choice supports); avoid side effects, as these are hard to capture in unit tests. Design your functions so that they can be run in isolation. Avoid multi-purpose functions. Avoid edge cases. Document what your functions are supposed to do.
• Write tests. Having unit tests means you know that your functions work for at least a subset of their inputs; it also means you have a sanity check to confirm your changes don't break anything. Make sure you understand the concepts of code coverage and input coverage, as well as the limitations of unit testing.
• Set up a 'workbench'. How exactly you do this depends on the language in question. Some languages, like Python or Haskell, come with an interactive interpreter, and you can load your existing code into it to play with it. This is perfect, since you can call your functions in any context you like, with minimal effort - an invaluable tool for finding and isolating bugs. Other languages don't have this luxury, and you'll have to resort to writing little interactive test programs.
• Write readable code. Make it a habit to write your code to express your intentions as clearly as possible. Document everything that isn't perfectly obvious.
• Write simple code. If your own brain has trouble understanding the entire codebase, then it's not simple, and it's highly unlikely someone else will be able to fully understand it. You can't debug code effectively unless you understand what it it supposed to do.
• Be easy on the 'Delete' button. Any code you don't need right now belongs in the trash can. If you need it later, revive it from source control (experience shows that this is extremely rare). The more code you dispose of, the smaller your debugging surface.
• Refactor early and often. Without refactoring, you cannot keep your code in a debuggable state while adding new features.

Answer 4

More Planning

It's inevitable that your going to spend a good chunk of time debugging, 10% is pretty a ambitious goal. Although one of the best ways to reduce time spent debugging and developing is spending more time in the planning phase.

This can range from diagrams, to pseudo code on a planning pad. Either way, you'll have more time to munch on what your planning on doing rather make those mistakes during development.

Answer 5

Work more carefully

This is the software equivalent of "measure twice cut once":

• Don't code if you're feeling distracted or tired.
• Spend enough time thinking about the problem such that you have a clean and elegant solution. Simple solutions are less likely to have problems.
• Give all your attention to the task. Focus.
• Read your code quickly after coding to try and look for mistakes. Self code review.
• Don't wait too long between coding and testing. Immediate feedback is important for improving.
• Avoid doing things that commonly lead to errors. Read on code smells.
• Pick the right tools for the job.

All that said, nothing is going to completely eliminate defects. You need to accept this as a fact of life. Given this fact plan for defects, e.g. unit test. Also don't take this to mean "take forever" (aka analysis-paralysis). It's about finding balance.

Answer 6

Other answers have already covered most of what I want to say, but I still want to to give you my (brutally honest) opinion anyway:

Basically, for non-trivial software work, expect to spend the overwhelming majority of your time on maintenance and debugging. If you're working on a mature, production software system, and you're spending less than 80-90% of your time on maintenance and debugging, you're doing well!

Now obviously, the distinction between "maintenance" and "debugging" is a bit subjective. Do you only consider "bugs" to be problems with the code that are found after it's released and users have complained about them? Or is it every slight little thing that goes wrong with your code once you've added something (found in your own pre-release testing phases)? In a non-trivial software system (depending on the usage patterns) one can be much bigger than the other. But in any case, this is what programming anything bigger than a toy "Hello world" program requires - lots and lots of maintenance and debugging. Some people even say something like "everything after the first line of code should be expected to be 'maintenance mode', with more refactoring and debugging effort than straight up new code writing".

TL;DR: It simply sounds to me like you might have a slightly unrealistic picture of what programming non-trivial software systems is all about. The vast majority of the effort is in finetuning, maintenance, refactoring, fixing bugs, and in general doing stuff that would come under "debugging" (maintenance) - at least in a very general sense - as opposed to doing totally fresh new work, writing fresh new code.

Answer 7

It's hard to give specific techniques without specific details on what you're doing and what technologies you're using. But even really good coders spend a lot of time testing and debugging.

A lot of writing good code without lots of bugs is experience. You make mistakes, then you fix them, then you remember what the mistakes were and what you had to do instead to do them right, and you don't make the same mistake the next time around. And if you're not even in college yet and you're already starting to think seriously about ways to make less mistakes, I'd say you're definitely ahead of the game.

Answer 8

CONTINUOUS INTEGRATION (CI) is the answer.

Continuous Integration = Configuration Management System (viz., Git, Mercurial, SVN,etc) + CI Tool + Unit Tests + Smoke Tests

That formula should propel you to read more on Continuous Integration (CI). Below are some resources in this area:

• XP Rules on Integrate Often
• Martin Fowler's Bliki on CI
• Continuous Integration: Improving Software Quality and Reducing Risk This one is an awesome book!

Answer 9

Really, to reduce debugging you can front load it by planning in greater depth. Haven't been to college yet? I think you will see in your mid to late college classes you will cover details of software development life cycle that very well may shine some light on your follies.

As I try to explain to my employers, the best way to reduce code maintenance and tech support is to spend the time to comprehensively plan your code in advance.

Answer 10

Test driven development can help reduce debugging time by:

• having lots of small, focused tests means if one fails there is only a small amount of code that could have caused the problem.
• working in small steps (by writing a failing test and then making it pass) means that you're able to focus on one task at a time. That is, making the current test past.
• refactoring after you make a test pass encourages you to keep your code clear and comprehensible - making it easier to follow if problems occur.

Even if you do use TDD, you will still have times when you need to use the debugger. When this happens, you should try to write a unit test to reproduce the scenario that caused the debugging session. This will ensure that if that issue ever occurs again it will be quickly caught when the test fails, and the test will act as a marker for the area of code that caused the problem - reducing the need for debugging.

Answer 11

Debug is inevitable in programming, but the key here is, is your code easy to debug or not? If you need to spend hours just to debug something simple, then there must be something really wrong with your code architecture.

You should get used to write clean code, and remove bad habits like copy pasting code and writing long methods, etc.

Beside, you should refactor your code from time to time. I suggest you to read Martin Fowler's book: Refactoring: Improving the Design of Existing Code

Answer 12

Others have mentioned testing and code review. These are both extremely useful but have a key difference - when is best to perform them. Testing is best done very close to originally writing the code, so you can more easily remember why you did things a certain way and can more quickly locate the problem when it fails the test. Code review, on the other hand, is better done a little bit later on. You want to have to look at the code without perfect recollection so that you don't gloss over details that you remember thinking about but didn't put in. You want to realize places where your code is not clear. You want the little extra effort of having to figure out what the code is doing. You want to be able to apply any new knowledge you have acquired about the problem or interactions with other code or new techniques. Basically, you want a change of perspective on the code.

All of this is still tangent to your question, though. In order to spend less time debugging, you have to understand why you had to debug in the first place. Misunderstanding the problem, imperfect knowledge of your tools and technologies, and just plain running into "the real data didn't match the sample data" types of issues are all going to manifest in different ways and need different techniques and types of practice to avoid in the future.

The last point I will make is experience. There is no easy way to get this, you just have to put in the time. As you gain experience you will spend less time debugging because you will write better code to begin with, notice problems earlier, and develop a better intuition for what the source of a problem might be. Keep at it and you will grow this steadily over your career.

Answer 13

Great answers above but no-one directly mentioned (although most hinted at this):

READ READ READ READ et at nauseam...

The more you know, the less you don't know. A bit cliche, but still the basic truth.

Once you have followed above tips and analytically documented the bugs, try to classify them, and then read the pertinent literature.

Was it a design decision issue? Read up on Design Patterns.

Was it a lack of knowledge of the framework or language? Bone up on that!

etc

There are two things a (live) developer can never escape: change (the only constant in IT) and RTFMing...

Answer 14

Unit tests and asserts

Where possible, factor your code into small pieces that can be tested in isolation. This isn't always practical, though. Some pieces of functionality depend on extremely complicated inputs. Some do something that can't easily be verified in an automated way, such as draw stuff to the screen. Sometimes non-determinism is involved, etc.

When you can't write good unit tests, the next best thing is asserts. While unit tests check to see if you get the right answer on some predetermined input, asserts check the sanity of intermediate steps on real-world inputs. If your code has bugs, it will fail quickly, close to the root of the problem and with a clear error message, rather than far from the problem with an ambiguous error message. Furthermore, asserts document assumptions and make your code more readable.

Answer 15

When you start a project, how many alternate approaches do you identify?

Do you have from two to four different approaches, with pros and cons for each? Do you then make a reasoned selection from among them?

Then, most important, do you weight simplicity as highly important?

The reason I ask is, in my experience, code volume, and thus the number of bugs (not to mention performance), can vary by more than an order of magnitude between one design approach and another. What I see highly experienced people doing is getting jobs done with no more code than necessary.

They are fully competent and aware of all the data structure algorithms, features of object-oriented languages, and so on, but their code looks as if they are not, because they use those things sparingly, or not at all, if the problem does not require them.

Answer 16

Every time you fix a bug, you want to avoid making the same mistake again. In order to do so, you can do the following:

• Write it down in a defect recording log, which includes:

  • defect type
  • phase in which the defect was injected
  • phase in which it was removed
  • fix time
  • a description of the problem and fix

• Adopt a styleguide to normalize the style of code you write

• Integrate secure coding rules into your code review process

• Visualize the control flow and data

References

• An Empirical Study of the Impact of PSP on Individual Engineers (pdf)

• Google CPP Style Guide

• Admitting Defeat On K&R in LCTHW – Zed A. Shaw

• IDE Plug-ins - A Security Testing Way to Automate Code Review | TCS Cyber Security Community

• Security Rules rule set for managed code

• Secure Coding in C++11 and C++14

• Visualize Java code execution

• SPARQL Visualization

• WWW2012 Tutorial Visualizing SPARQL Queries