27

One of the few things that most software developers agree on is that you shouldn't rely on code to work correctly unless you test it. If you don't test it, it may have hidden bugs that are only going to cause you more work down the road.

I understand how to test my normal code, but how should I test my test code to make sure it can effectively find and report errors when they are present? I personally have been stupid enough to write erroneous test cases that would pass when they should not have, thus defeating the purpose of my writing tests in the first place. Fortunately, I found and fixed the errors in time, but according to testing mantra it seems like no test suite would be complete without having its own set of tests to make sure it worked.

It seems to me that the best way to do this would be to make sure the test fails for buggy code.* If I spend 2 minutes alternately adding bugs to the code and making sure it fails, I should have an acceptable degree of confidence that the tests 'work'. This brings me to my second question: What are good ways to introduce bugs to make sure that they are caught by the test cases? Should I just randomly comment out statements, make sure the wrong branch of an if-else gets run by negating its condition, and change the execution order of code with side-effects, etc., until I'm satisfied my tests will catch most common bugs? How do professional developers validate that their tests actually do what they're supposed to do? Do they just assume the tests work, or do they take the time to test them as well? If so how do they test the tests?

I'm not suggesting the people should spend so much time testing their tests and then testing the tests for their tests that they never actually write the real code, but I've done stupid enough things that I feel like I could benefit from a bit of 'meta-testing', and was curious about the best way to go about it. :D

* I could check to see if the test passes when testing 'bug-free' code, but using the code as a spec for the test seem quite backwards...

Improve this question
3
  • 1
    Sounds like you don't have confidence in your unit tests - most likely because you lack a lot of experience in writing tests? In that case it would be unreasonable to write more tests and expect a different outcome. Just keep doing what you're doing, be as thorough as you can (test for failure as well as success) and soon your unit tests will start to pay for themselves - and your confidence in them will grow.
    – MattDavey
    Jul 24, 2012 at 7:08
  • Possible duplicate of How to test the tests?
    – gnat
    Sep 9, 2017 at 18:45
  • Before using more tools, use your actual tools better. Write less tests, but more efficient and better written tests. You cannot trust something you don’t understand.
    – Steve Chamaillard
    Jan 19, 2019 at 10:19

8 Answers 8

Reset to default
21

The standard flow for TDD is:

  1. Write a failing test. (Red)
  2. Make the smallest code change that makes it pass (Green)
  3. Refactor (Keeping it green)

The test for your tests in this case is step 1 - making sure that the test fails before you make any code changes.

Another test that I like is whether you can delete some code and re-implement it a different way, and your tests fail after deletion but work with a different algorithm in place.

As with all things, there is no magic bullet. Forgetting to write a required test is just as easy for a developer to do as forgetting to write the code. At least if you're doing both, you have twice as many opportunities to discover your omission.

Improve this answer
7
  • 5
    Double bookkeeping: The unit tests test the production code and vice versa. They are two different ways to state the same problem. Like in double bookkeeping, where you record your transactions in two different ways and both must get the same totals.
    – EricSchaefer
    Jul 24, 2012 at 18:21
  • The question is about the problem when step 2 makes the test green even though the code is still buggy. Simple example: you have a function which must return a pointer to a non-empty list object. Your test tests whether the pointer is null and fails at step 1. You make the smallest code change that makes it pass by returning an empty list. The test is now "green" because you have two bugs.
    – Christian Hackl
    Jan 21, 2019 at 6:00
  • @ChristianHackl at that development stage it is a perfect implementation! You have to add one (or more) tests that fail at first to further specify the expected behavior. Subsequently you implement it, making these tests green.
    – Andy
    Jan 22, 2019 at 9:23
  • @Andy: Care to elaborate how this is a perfect implementation when I have a bug both in the code and in the test, and one keeps me from noticing the other? (The bug in the code is that an empty list is returned, and the bug in the test is that I check for null and not for empty.)
    – Christian Hackl
    Jan 22, 2019 at 12:30
  • 1
    @Andy: "When you translate your requirements to tests, step by step, there is little room for bugs." - If it was so easy, then you'd just do the same careful translation to the code itself to minimise room for bugs, and you wouldn't need tests anymore. The point is that the TDD approach has nothing to say about the problem asked in the question (how to test the tests), because a successful test does not mean that the code isn't buggy.
    – Christian Hackl
    Jan 23, 2019 at 9:11
11

One approach is Mutation Testing, using a tool like Jester:

Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed

Improve this answer
4

Tests for tests? Don't go that road. Then you'll probably need tests for tests for tests, and then tests for tests for tests for tests... where do you stop?

Usual testing flow goes like this, and as a developer, you'll spend majority of your time on points 1-3:

  1. Code
  2. Unit tests
  3. Integration tests
  4. System/other automated
  5. QA/human testers

If I spend 2 minutes alternately adding bugs to the code (...)

Your code will eventually "grow" its own bugs, don't waste time introducing them by hand. Not to mention, is a thing you knew about upfront really a bug? Bugs will come, I wouldn't worry about that.

Should I just randomly comment out statements, make sure the wrong branch of an if-else gets run by negating its condition (...)

This is actually a viable approach to verify whether you actually test what you think you do. I don't think it is always that good as it suffers from the same problem as "test for test for test ..." thing: when do you stop altering code knowing code you're testing 100% works?

It's also good to remember about all time classic pragmatic programmer advice - you ain't gonna need it. Be agile, write tests and code for actual bugs, instead for those hypothetical that might or might not appear.

Improve this answer
2
  • 5
    I'm not worried about my code growing its own bugs, I'm worried about my tests catching them when they happen. If my tests are faulty, they won't do their job and I'll think that I've got rid of a certain class of bug when they actually still exist, just making my job harder because I'm looking at inaccurate test results (pass when they should fail).
    – Gordon Gustafson
    Jul 23, 2012 at 21:49
  • 1
    @CrazyJugglerDrummer: your tests won't catch all the bugs that's for sure. They don't serve that purpose - this is where QA comes in. If they would do so, it would mean software is bug-free, unless source code changes, which I've never seen.
    – k.m
    Jul 23, 2012 at 21:54
3

By construction, functional code and test code are tested one against the other. One problem remains: the case of common mode bugs, when a bug in functional code is hidden by a bug in test code. TDD is not immune to this effect. This is why testing is usually performed at multiple levels by different people in order to decrease this probability.

Improve this answer
1
  • So don’t copy code from the implementation into the test, and best is to let someone else write the test. They are just as likely as you to write buggy tests, but less likely to write tests that match the bugs in your implementation.
    – gnasher729
    Jan 22 at 12:16
0

There is a mutation testing that evaluates and measures the suitability and quality of the test.

We can use this to evaluate "the test" itself.

In brief, We can evaluate our test (e.g. TestA) by testing TestA can find the difference between the code and its mutation codes(very similar but slightly different code with original code).

If TestA cannot find the difference between the code and its mutation codes, It means that the TestA has too rough regulations to test the original code.

Improve this answer
0
0

You test your unit test once when you write it, in the debugger. Then you leave it alone and forget about it. There is no problem here.

Consider this. What is the purpose of a unit test? It notifies you when any of the numerous changes you will make in your main program accidentally changes the logic in that program. You want to have that because you know that any change potentially breaks something. Which is exactly why there is no problem if you do not test your test: you do not mess with your test until you purposely change the logic of your program (which would require you to revisit the test and test it once more), so your test is not likely to break accidently.

Improve this answer
0

Maybe a slightly different question, but...

You might have more elaborate test setup for integration tests or if you're in a larger project / software engineering org you might have a library of test fixtures, test tools, etc. There it might make sense to test your test fixtures and test tools by creating tiny test suites where you use those things. pytest has a neat example of how you might go about this in pytester.

Improve this answer
0

Especially for new tests, a test failure doesn’t mean “my code failed” but “my code and my test disagree”. There’s no fairy godmother telling you which is wrong, so you check them both. And fix the one that is faulty, and sometimes both.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.