How to get better at testing your own code

Question

I am a relatively new software developer, and one of the things I think I should improve is my ability to test my own code. Whenever I develop a new functionality, I find it really difficult to follow all the possible paths so I can find bugs. I tend to follow the path where everything works. I know this is a well known issue that programmers have, but we don't have testers at my current employer and my colleagues seem to be pretty good at this.

At my organization, we do neither test-driven development nor unit testing. It would help me a lot but it's not likely that this will change.

What do you guys think I could do in order to overcome this? What approach do you use when testing your own code?

Answer 1

The job of a coder is to build things.

The job of a tester is to break things.

The most difficult is to break things you just built. You will succeed only by going over this psychological barrier.

Answer 2

Franciso, I'm going to make some assumptions here, based on what you've said:

"We don't do neither TDD nor unit testing. It would help me a lot but it's not likely that this will change."

From this, I suspect your team does not place a lot of value on testing or management will not budget time for the team to try and tidy up existing code and keep technical debt to a minimum.

Firstly, you need to convince your team/management the value of testing. Be diplomatic. If management is keeping your team moving forward, you need to show them some facts, such as the defect rate for every release. Time spent fixing defects could be better spent on other things, such as improving the application and making it more adaptive to future requirements.

If the team and management in general are apathetic about fixing the code and you feel unhappy about it, you may need to look for another place to work, unless you can convince them as I have said. I've encountered this problem to different degrees in all places I've worked. It could be anything from lack of a proper domain model, to poor communication in the team.

Caring about your code and the quality of the product you develop is a good attribute and one you always want encourage in other people.

Answer 3

If you code in C, Objective-C or C++ you can use the CLang Static Analyzer to critique your source without actually running it.

There are some memory debugging tools available: ValGrind, Guard Malloc on Mac OS X, Electric Fence on *NIX.

Some development environments provide the option to use a debugging memory allocator, that does stuff like fill newly allocated pages and newly freed pages with garbage, detect the freeing of unallocated pointers, and write some data before and after each heap block, with the debugger being called if the known pattern of that data ever changes.

Some guy on Slashdot said he got a lot of value out of single-stepping ever new line of source in a debugger. "That's it" he said. I don't always follow his advice, but when I have it has been very helpful to me. Even if you don't have a test case that stimulates an uncommon code path, you can twiddle a variable in your debugger to take such paths, say by allocating some memory, then using the debugger to set your new pointer to NULL instead of the memory address, then stepping through the allocation failure handler.

Use assertions - the assert() macro in C, C++ and Objective-C. If your language doesn't provide an assert function, write one yourself.

Use asserts liberally, then leave them in your code. I call assert() "The test that keeps on testing". I use them most commonly to check preconditions at the entry point of most of my functions. That's one part of "Programming by Contract", which is built into the Eiffel programming language. The other part is postconditions, that is, using assert() at function return points, but I find that I don't get as much mileage out of that as preconditions.

You can also use assert to check class invariants. While no class is strictly required to have any invariant at all, most sensibly designed classes do have them. A class invariant is some condition that is always true other than inside of member functions that might temporarily place your object into an inconsistent state. Such functions always must restore the consistency before they return.

Thus every member function could check the invariant upon entry and exit, and the class could define a function called CheckInvariant that any other code could call at any time.

Use a code coverage tool to check which lines of your source are actually getting tested, then design tests that stimulate the un-tested lines. For example you could check low memory handlers by running your app inside a VM that is configured with little physical memory, and either no swap file or a very small one.

(For some reason I was never privy to, while the BeOS could run without a swap file, it was highly unstable that way. Dominic Giampaolo, who wrote the BFS filesystem, urged me never to run the BeOS without swap. I don't see why that should matter, but it must have been some kind of implementation artifact.)

You should also test your code's response to I/O errors. Try storing all your files on a network share, then disconnect your network cable while your app has a high workload. Similarly disconnect the cable - or turn off your wireless - if you are communicating over a network.

One thing that I find particularly infuriating are websites that don't have robust Javascript code. Facebook's pages load dozens of little Javascript files, but if any one of them fails to download, the whole page breaks. There just has to be some way either to provide some fault tolerance, say by retrying a download, or to provide some kind of reasonable fallback when some of your scripts didn't download.

Try killing your app with the debugger or with "kill -9" on *NIX while it is right in the middle of writing a big, important file. If your app is well-architected, the entire file will get written or will not be written at all, or maybe if it is only partially written, what does get written will not be corrupted, with what data that is saved being completely usable by the app upon re-reading the file.

databases always have fault-tolerant disk I/O, but hardly any other kind of app does. While journaled filesystems prevent filesystem corruption in the event of power failure or crashes, they don't do anything at all to prevent the corruption or loss of end-user data. That is the responsibility of the user applications, but hardly any other than databases implement fault-tolerance.

Answer 4

When I look at testing my code, I usually go through a series of thought processes:

1. How do I break this "thingy" down into a testable size chunks? How can I isolate just what I want to test? What stubs/mocks should I create?
2. For each chunk: How do I test this chunk to make sure that it responds correctly to a reasonable set of correct inputs?
3. For each chunk: How do I test that the chunk responds correctly to incorrect inputs (NULL pointers, invalid values)?
4. How do I test for boundaries (e.g. where values go from signed to unsigned, 8 bit to 16 bit, etc.)?
5. How well do my tests cover the code? Are there any conditions I missed? [This is a great place for code-coverage tools.] If there is code that was missed and can never be executed, does it really need to be there? [That's a whole other question!]

The easiest way I have found to do this is to develop my tests along with my code. As soon as I have written even a fragment of code, I like to write a test for it. Trying to do all of the testing after having coded several thousand lines of code with non-trivial cyclomatic code complexity is a nightmare. Adding one or two more tests after adding a few lines of code is really easy.

BTW, just because the company you work at and/or your colleagues don't do Unit Testing or TDD, doesn't mean you can't try them, unless they are specifically prohibited. Perhaps using them to create robust code will be a good example for others.

Answer 5

In addition to the advice given in the other answers, I would suggest using static analysis tools (Wikipedia has a list of a number of static analysis tools for various languages) to find potential defects before testing begins as well as monitoring some metrics that relate to the testability of code, such as cyclomatic complexity, the Halstead complexity measures, and cohesion and coupling (you can measure these with fan-in and fan-out).

Finding hard to test code and making it easier to test will, well, make it easier for you to write test cases. Also, catching defects early will add value to your entire quality assurance practices (which include testing). From here, getting familiar with unit testing tools and mocking tools will make it easier for you to implement your testing.

Answer 6

You could look into the possible usage of Truth Tables to help you define all of the potential paths in your code. It's impossible to account for all possibilities in complex functions, but once you have your handling established for all known paths, you can establish a handling for the else case.

Most of this particular ability is learned by experience though. After you've used a certain framework for a significant amount of time, you begin to see the patterns and earmarks of behavior that will let you look at a piece of code and see where a small change could cause a major error. The only way I can think of to increase your aptitude in this is practice.

Answer 7

If like you said you don't need Unit testing I don't see any better approach than trying to break your own code manually.

Try to push your code to the limits. For example try to pass variables to a function exceeding the boundary limits. Do you have a function which is supposed to filter user input? Try to input different combinations of characters.

Consider the user's point of view. Try to be one of the users who will use your application or library of functions.

Answer 8

but we don't have testers at my current employer and my colleagues seem to be pretty good at this

Your colleagues must be truly exceptional to not follow TDD or unit test and never generate bugs so on some level I doubt that they are not performing any unit testing themselves.

I am guessing that your colleagues are doing more testing than is being let on but because this fact isn't known by management then the organization suffers as a result because management gets the impression that true testing is not being performed and bug numbers are low thus testing is unimportant and time will not be scheduled for it.

Talk to your colleagues and try to figure out what kind of unit testing they are doing and emulate that. At a later time you can prototype better ways to unit test and TDD attributes and slowly introduce these concepts to the team for easier adoption.

Answer 9

• Write your tests before you write your code.
• Any time you fix a bug that wasn't caught by a test, write a test to catch that bug.

You should be able to get coverage on what you write even if your organization doesn't have full coverage. Like so many things in programming, the experience of doing it again and again one of the best ways to be efficient at it.

Answer 10

In addition to all the other comments, since you say your colleagues are good at writing non-happy-path tests, why not ask them to pair with you on writing some tests.

The best way to learn is be seeing how it's done and pulling in what you learn from that.

Answer 11

Black box testing! You should be creating your classes/methods with testing in mind. Your tests should be based on the specification of the software and should be clearly defined on your Sequence diagram (via use cases).

Now since you may not want to do test driven development...

Put input validation on all incoming data; don't trust anyone. The .net framework has throws a lot of exceptions based on invalid arguments, null references and invalid states. You should already be thinking of using input validation on the UI layer so it's the same trick in the middle-ware.

But you should really be doing some sort of automated testing; that stuff saves lives.

Answer 12

In my experience

Test unit, if it is not completely automatic,it is useless. It is more like a Pointy Haired Boss could buy. Why?, because Test Unit promised you to save time (and money) automating some test process. But, some test unit tools do the opposite, it force programmers to work in some weird fashion and force other to create over-extend test. Most of the time, it will not save working hour but increase moving time from QA to developer.

UML is another time waster. a single whiteboard + pen could do the same, cheaper and quickly.

BTW,How to be good at coding (and avoid bugs) ?

• a) atomicity. One function that do one simple (or a few single task). Because it is easy to understand, it is easy to track and it is easy to solve it.

• b) Homology. If, for example, you call a database using a store procedure then do for the rest of the code.

• c) Identify,reduce and isolate "creative code". Most of the code is pretty much copy & paste. Creative code is the opposite, a code that is new and it act as a prototype, it can fail. This code is prone of logic bug, so it is important to reduce, isolate and identify it.

• d) "Thin ice" code, is the code that you know that it is "incorrect" (or potentially dangerous) but still is needing, for example unsafe code for a multi task process. Avoid if you can.

• e) Avoid black box code, this include code that it is not done by you (for example framework) and expression regular. It is easy to miss a bug with this kind of code. For example, i worked in a project using Jboss and i found not one but 10 error in Jboss (using the latest stable version), it was a PITA to find those. Avoid specially Hibernate, it hide the implementation, hence the bugs.

• f) add comments in your code.

• g) user input as source of bugs. identify it. For example, SQL Injection is caused by a user input.

• h) Identify bad element of the team and separate the task assigned. Some programmers are prone to screw the code.

• i) Avoid unnecesary code. If, for example, the class need Interface then use it, otherwise avoid to add irrelevant code.

a) and b) are key. For example, i had a problem with a system, when i clicked in a button (save) it didn't save the form. Then i did a checklist:

• the button works?... yes.
• the database store something?. no, so the error was in a middle step.
• then, the class that store in the database work?. no <-- ok, i found the error. (it was related with database permission). Then i checked not only this procedure but every procedure that do the same (because the homology of the code). It took me 5 minutes to track the bug and 1 minute to solve it (and many other bugs).

And a sidenote

Most of the time QA suck (as a separate section of Dev), it is useless if they are not worked in the project. They do some generic test and nothing else much. They are unable to identify most logic bugs. In my case, i was working in a prestigious bank, a programmer finished a code then send it to QA. QA approved the code and was put in production... then the code failed (an epic fail), do you know who was blamed?. yes, the programmer.

Answer 13

A tester and a programmer face the problem from different angles, but both roles should fully test functionality and find bugs. Where the roles differ is in focus. A classical tester sees the application from only the outside (i.e. black box). They are experts on the functional requirements of the app. A programmer is expected to be an expert on both the functional requirements and the code (but tends to focus more on the code).

(It depends on the organization whether programmers are explicitly expected to be an expert on requirements. Regardless, the implicit expectation is there -- if you design something wrong, you -- not the requirements person -- receive the blame.)

This dual expert role is taxing on the mind of the programmer and, excepting the most experienced, can decrease proficiency in requirements. I find that I must mentally shift gears to consider the users of the application. Here is what helps me:

1. Debugging; set breakpoints in the code and run the application. Once you hit a breakpoint, step through the lines as you interact with the application.
2. Automated testing; write code that tests your code. This only helps at tiers below the UI.
3. Get to know your testers; they may know the application better than you do, so learn from them. Ask them what the weaknesses of your application are, and what tactics they use to find bugs.
4. Get to know your users; learn to walk in your users' shoes. The functional requirements are the fingerprint of your users. There are often many things your users know about the application that may not come through clearly in the functional requirements. As you understand your users better -- the nature of their work in the real world and how your application is supposed to help them -- you'll understand better what the application is supposed to be.

Answer 14

I think you want to work on two fronts. One is political, getting your organization to adopt testing at some level (with the hope that over time they will adopt more). Talk to QA engineers outside your workplace. Find lists of QA books. Poke around relevant wikipedia articles. Familiarize yourself with QA principles and practices. Learning this stuff will prepare you to make the most convincing case you can in your organization. Good QA departments do exist and they do add considerable value to their organizations.

As an individual developer, adopt strategies to use in your own work. Use TDD yourself by co-developing code and tests. Keep tests clear and well maintained. If asked why you're doing this, you can say you are preventing regressions and it keeps your thought process better organized (both of which will be true). There is an art to writing testable code, learn it. Be a good example for your fellow developers.

Partly I'm preaching to myself here, because I do way less of this stuff than I know I should.