22

One of the few things that most software developers agree on is that you shouldn't rely on code to work correctly unless you test it. If you don't test it, it may have hidden bugs that are only going to cause you more work down the road.

I understand how to test my normal code, but how should I test my test code to make sure it can effectively find and report errors when they are present? I personally have been stupid enough to write erroneous test cases that would pass when they should not have, thus defeating the purpose of my writing tests in the first place. Fortunately, I found and fixed the errors in time, but according to testing mantra it seems like no test suite would be complete without having its own set of tests to make sure it worked.

It seems to me that the best way to do this would be to make sure the test fails for buggy code.* If I spend 2 minutes alternately adding bugs to the code and making sure it fails, I should have an acceptable degree of confidence that the tests 'work'. This brings me to my second question: What are good ways to introduce bugs to make sure that they are caught by the test cases? Should I just randomly comment out statements, make sure the wrong branch of an if-else gets run by negating its condition, and change the execution order of code with side-effects, etc., until I'm satisfied my tests will catch most common bugs? How do professional developers validate that their tests actually do what they're supposed to do? Do they just assume the tests work, or do they take the time to test them as well? If so how do they test the tests?

I'm not suggesting the people should spend so much time testing their tests and then testing the tests for their tests that they never actually write the real code, but I've done stupid enough things that I feel like I could benefit from a bit of 'meta-testing', and was curious about the best way to go about it. :D

* I could check to see if the test passes when testing 'bug-free' code, but using the code as a spec for the test seem quite backwards...

  • Sounds like you don't have confidence in your unit tests - most likely because you lack a lot of experience in writing tests? In that case it would be unreasonable to write more tests and expect a different outcome. Just keep doing what you're doing, be as thorough as you can (test for failure as well as success) and soon your unit tests will start to pay for themselves - and your confidence in them will grow. – MattDavey Jul 24 '12 at 7:08
  • Possible duplicate of How to test the tests? – gnat Sep 9 '17 at 18:45
  • Before using more tools, use your actual tools better. Write less tests, but more efficient and better written tests. You cannot trust something you don’t understand. – Steve Chamaillard Jan 19 at 10:19
16

The standard flow for TDD is:

  1. Write a failing test. (Red)
  2. Make the smallest code change that makes it pass (Green)
  3. Refactor (Keeping it green)

The test for your tests in this case is step 1 - making sure that the test fails before you make any code changes.

Another test that I like is whether you can delete some code and re-implement it a different way, and your tests fail after deletion but work with a different algorithm in place.

As with all things, there is no magic bullet. Forgetting to write a required test is just as easy for a developer to do as forgetting to write the code. At least if you're doing both, you have twice as many opportunities to discover your omission.

  • 4
    Double bookkeeping: The unit tests test the production code and vice versa. They are two different ways to state the same problem. Like in double bookkeeping, where you record your transactions in two different ways and both must get the same totals. – EricSchaefer Jul 24 '12 at 18:21
  • The question is about the problem when step 2 makes the test green even though the code is still buggy. Simple example: you have a function which must return a pointer to a non-empty list object. Your test tests whether the pointer is null and fails at step 1. You make the smallest code change that makes it pass by returning an empty list. The test is now "green" because you have two bugs. – Christian Hackl Jan 21 at 6:00
  • @ChristianHackl at that development stage it is a perfect implementation! You have to add one (or more) tests that fail at first to further specify the expected behavior. Subsequently you implement it, making these tests green. – Andy Jan 22 at 9:23
  • @Andy: Care to elaborate how this is a perfect implementation when I have a bug both in the code and in the test, and one keeps me from noticing the other? (The bug in the code is that an empty list is returned, and the bug in the test is that I check for null and not for empty.) – Christian Hackl Jan 22 at 12:30
  • @ChristianHackl you are right according the empty check. That, indeed, should be done in a test as well. When you translate your requirements to tests, step by step, there is little room for bugs. Except those when you don't stick to the spec (like overlooking a not-empty check). – Andy Jan 22 at 18:48
9

One approach is Mutation Testing, using a tool like Jester:

Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed

5

Tests for tests? Don't go that road. Then you'll probably need tests for tests for tests, and then tests for tests for tests for tests... where do you stop?

Usual testing flow goes like this, and as a developer, you'll spend majority of your time on points 1-3:

  1. Code
  2. Unit tests
  3. Integration tests
  4. System/other automated
  5. QA/human testers

If I spend 2 minutes alternately adding bugs to the code (...)

Your code will eventually "grow" its own bugs, don't waste time introducing them by hand. Not to mention, is a thing you knew about upfront really a bug? Bugs will come, I wouldn't worry about that.

Should I just randomly comment out statements, make sure the wrong branch of an if-else gets run by negating its condition (...)

This is actually a viable approach to verify whether you actually test what you think you do. I don't think it is always that good as it suffers from the same problem as "test for test for test ..." thing: when do you stop altering code knowing code you're testing 100% works?

It's also good to remember about all time classic pragmatic programmer advice - you ain't gonna need it. Be agile, write tests and code for actual bugs, instead for those hypothetical that might or might not appear.

  • 3
    I'm not worried about my code growing its own bugs, I'm worried about my tests catching them when they happen. If my tests are faulty, they won't do their job and I'll think that I've got rid of a certain class of bug when they actually still exist, just making my job harder because I'm looking at inaccurate test results (pass when they should fail). – Gordon Gustafson Jul 23 '12 at 21:49
  • @CrazyJugglerDrummer: your tests won't catch all the bugs that's for sure. They don't serve that purpose - this is where QA comes in. If they would do so, it would mean software is bug-free, unless source code changes, which I've never seen. – k.m Jul 23 '12 at 21:54
3

By construction, functional code and test code are tested one against the other. One problem remains: the case of common mode bugs, when a bug in functional code is hidden by a bug in test code. TDD is not immune to this effect. This is why testing is usually performed at multiple levels by different people in order to decrease this probability.

0

You test your unit test once when you write it, in the debugger. Then you leave it alone and forget about it. There is no problem here.

Consider this. What is the purpose of a unit test? It notifies you when any of the numerous changes you will make in your main program accidentally changes the logic in that program. You want to have that because you know that any change potentially breaks something. Which is exactly why there is no problem if you do not test your test: you do not mess with your test until you purposely change the logic of your program (which would require you to revisit the test and test it once more), so your test is not likely to break accidently.

-2

There is a mutation testing that evaluates and measures the suitability and quality of the test.

We can use this to evaluate "the test" itself.

In brief, We can evaluate our test (e.g. TestA) by testing TestA can find the difference between the code and its mutation codes(very similar but slightly different code with original code).

If TestA cannot find the difference between the code and its mutation codes, It means that the TestA has too rough regulations to test the original code.

protected by gnat Jan 17 at 7:12

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.