58

We test our code to make it more correct (actually, less likely to be incorrect). However, the tests are also code -- they can also contain errors. And if your tests are buggy, they hardly make your code better.

I can think of three possible types of errors in tests:

  1. Logical errors, when the programmer misunderstood the task at hand, and the tests do what he thought they should do, which is wrong;

  2. Errors in the underlying testing framework (eg. a leaky mocking abstraction);

  3. Bugs in the tests: the test is doing slightly different than what the programmer thinks it is.

Type (1) errors seem to be impossible to prevent (unless the programmer just... gets smarter). However, (2) and (3) may be tractable. How do you deal with these types of errors? Do you have any special strategies to avoid them? For example, do you write some special "empty" tests, that only check the test author's presuppositions? Also, how do you approach debugging a broken test case?

Improve this question
6
  • 3
    Every introductory piece I've read about mocking seems to hit this problem. Once you start mocking things the tests always seem to be more complicated than the code they're testing. Obviously this is less likely to be the case when testing real-world code, but its quite disheartening when you're trying to learn.
    – Carson63000
    Oct 13, 2010 at 3:52
  • @Carson63000 If it's a simple test testing something with a tested mock, the complexity is split and under control (, I think).
    – mlvljr
    Nov 12, 2010 at 10:43
  • 20
    But then how do you test the test tests?
    – ocodo
    Feb 20, 2011 at 21:30
  • +1. Item 1 could be a requirements error. Can only be prevented by reviewing the requirements. Probably out of the programmer's hands unless they are also the requirements analyst
    – MarkJ
    Feb 27, 2012 at 17:30
  • 2
    @ocodo: The same way you watch the Watchers. :)
    – Greg Burghardt
    Mar 13, 2017 at 13:33

12 Answers 12

Reset to default
21

The tests are already tested. Tests are by design protected from the bugs, because the testing only detects differences between code and our expectations. If there are problems we have an error. The error could be in the code or in tests.

There are some techniques that prevents you from adding the same bug in both your code and tests:

  1. The client should be different person than the implementer.

  2. First write the tests and then the code (like in Test Driven Development).

  3. You don't need to test the underlying platform. The tests not only exercise the code written by you, but they run the code from the platform too. While you don't necessary want to catch bugs in the testing platform, is very hard to write code and tests that always hides a bug in the platform, in other words is very hard to have a systematic bug in both your tests/code and in the platform, and the probability is lowered with each test that you create. Even if you would try to do this you would have a very hard task.

  4. You could have bugs in tests but usually they are caught easily because the tests are tested by the code developed. Between the code and the tests you have a self enforcement feedback. Both make prediction about how a specific call of an interface should behave. If the response is different you don't necessary have a bug in the code. You could have a bug in the test as well.

Improve this answer
2
  • Very nice answer. I like the idea of a self-reinforcing loop between the tests and the code and the observation that it would be difficult to write tests consistently hiding bugs in the underlying platform.
    – Ryszard Szopa
    Oct 13, 2010 at 0:15
  • which doesn't prevent tests from being created based on flawed assumptions about what the actual code should do. Which can lead to very nasty bugs remaining undetected during testing. The only way to prevent that is to have the tests written by a third party with no relation at all to the organisation writing the actual code, so they can't "pollute" each other's thinking when it comes to interpreting the requirements documents.
    – jwenting
    Mar 14, 2017 at 13:13
26

Try making the individual tests as small (short) as possible.

This should reduce the chances of creating a bug in the first place. Even if you manage to create one, it's easier to find. Unit tests are supposed to be small and specific, with low tolerance for failure and deviation.

In the end, it's probably just a matter of experience. The more tests you write, the better you become at it, the less chance you have to make crappy tests.

Improve this answer
4
  • 6
    What if the tests need some rather complicated setup? Sometimes these sort of things are not under your control.
    – Ryszard Szopa
    Oct 12, 2010 at 23:38
  • Well, I'm guessing the complicated setup are "initial condition" of the tests. If that fails, all your tests should fail. In fact, I'm working on such a project right now, and people who never used unit tests constantly asked the same thing..until we explained what unit tests really are :) Then they realized it can be done, in spite of tremendous complexity of the project.
    – dr Hannibal Lecter
    Oct 13, 2010 at 7:21
  • 1
    What's the best way to check that this "initial condition" is met is exactly the point of my question. Do you write a separate test for that? Or just assume that the tests will break if this condition is not true? What about the situation when the setup is not "catastrophically" bad, just slightly off?
    – Ryszard Szopa
    Oct 13, 2010 at 14:52
  • 2
    Your tests should fail if the initial conditions are not right, that's the whole point. When in state A, you expect result B. If you don't have state A, a test should fail. At that point you can investigate why it failed, bad initial conditions or a bad test, but it should fail in both cases. Even if it is, as you say, "slightly off" (i.e. "A" => "B", "a" => "b", but never "a" => "B" or your test is bad).
    – dr Hannibal Lecter
    Oct 13, 2010 at 15:38
21

One tactic is to write the test before the code it tests, and ensure the test fails first for the right reason. If you use TDD you should get at least this level of testing of tests.

A more exhaustive way to test the quality of a test suite is to use mutation testing.

Improve this answer
4
  • 2
    And that your test fails for the right reason.
    – Frank Shearar
    Oct 13, 2010 at 4:33
  • @Frank - Yes. I'll add that to the answer.
    – Don Roby
    Oct 13, 2010 at 9:35
  • And you're adding a new test for the new behavior to be tested. Don't add to existing tests.
    – Huperniketes
    Oct 14, 2010 at 17:00
  • @DonRoby, Have you found mutation testing useful in practice? What deficiencies you've found in your test cases with that?
    – dzieciou
    Nov 25, 2012 at 21:42
4

For #1 and #3: Unit tests should not contain any logic, if you do then you are probably testing more than one thing in your unit test. One best practice for unit testing is to only have one test per unit test.

Watch this video by Roy Osherove to learn more on how to write unit tests well.

Improve this answer
2
  • 1
    ad #3 - I agree that the tests should be as simple as possible, and should not contain any logic. However, think about the setup phase of the test, when you create the objects that it will need. You may create slightly wrong objects. This is the kind of problems I am thinking about.
    – Ryszard Szopa
    Oct 13, 2010 at 0:57
  • When you say 'slightly wrong objects' do you mean the object state is not correct or the actual design of the object is not correct? For object state you could probably write tests to check it's validity. If the design is wrong then the test should fail.
    – Piers Myers
    Oct 13, 2010 at 1:08
4

Hey.
You have to applications:

  • Your product
  • Your test for that product.

When you are running tests against your product, you actually are not intrested in test itself, but in interaction between your product and your tests. If test fails it doesn't say that application has a bug. It says that interaction between product and test was not successful. Now it is your job to determine what went wrong. It can be either:

  • application is not behaving as you expected (this expectation is expressed in your test)
  • application is behaving correctly, you just haven't documented this behavior correctly (in your tests)

For me tests failing are not simple feedback, that this and that is wrong. It is indicator that there is inconsistency, and I need to examine both to check want went wrong. In the end I am responsible for verifying that application is correct, tests are just a tool to highlight areas that may worth checking.

Tests are only checking some parts of application. I test the application, I test the tests.

Improve this answer
3

In terms of #1 - I think it's a good idea to pair / code review for this side of things. It's easy to make presuppositions or just get things wrong but if you have to explain what your test is doing, what the point is, you're more likely to pick up if you're aiming at the wrong target.

Improve this answer
3

Tests shouldn't be "smart" enough to harbour bugs.

The code you're writing implements a set of specifications. (If X then Y, unless Z in which case Q, etc etc). All the test should be attempting to accomplish is to determine that X really is Y unless Z in which case Q. This means all a test should be doing is setting X and verifying Y.

But that doesn't cover all cases, you are probably saying, and you'd be right. But if you make the test "smart" enough to know that X should only by Y if not Z then you're basically re-implementing the business logic in the test. This is problematic for reasons we'll go into a bit more deeply below. You shouldn't improve the code coverage by making your first test "smarter", you should instead add a second dumb test which sets X and Z and verifies Q. That way you'll have two tests, one that covers the general case (sometimes also known as the happy path) and one that covers the edge case as a separate test.

There are a number of reasons for this, first and foremost is how do you determine if a failed test is due to a bug in the business logic or a bug in the tests? Obviously the answer is that if the tests are as simple as possible they're very unlikely to be harbouring bugs. If you think your tests need testing then you're testing wrong.

Other reasons include that you're just replicating effort (as I already mentioned, writing a test smart enough to exercise all possibilities in a single test is basically replicating the business logic you're trying to test in the first place), if requirements change then the tests should be easy to change to reflect the new requirements, tests serve as a sort of documentation (they're a formal way of saying what the specification of the unit under test is), and so on.

TL:DR: If your tests need testing you're doing it wrong. Write dumb tests.

Improve this answer
2

There must be a point when one should stop trying to unit test. Should know when to draw the line. Should we write test cases to test test cases? What about the new test cases written to test test cases? How will we test them?

if (0 > printf("Hello, world\n")) {
  printf("Printing \"Hello, world\" failed\n");
}

Edit: Updated with explanation as suggested by comment.

Improve this answer
5
  • -1 What? This seems to have no relevance.
    – alternative
    Oct 12, 2010 at 23:38
  • 2
    There must be a point when one should stop trying to unit test. Should know when to draw the line. Should we write test cases to test test cases? What about the new test cases written to test test cases? How will we test them?
    – aufather
    Oct 12, 2010 at 23:52
  • 2
    Process Brain raised EInfiniteRecursion while attempting to extrapolate your statement...
    – Mason Wheeler
    Oct 13, 2010 at 0:10
  • Replace your answer with your comment and you'll get a +1
    – Note to self - think of a name
    Oct 13, 2010 at 0:13
  • 3
    In all fairness, your example is a straw man. You're testing the printf() subsystem in a C library, not the actual program calling printf(). I do agree, however, that at some point one must break the recursive testing of tests.
    – Tim Post
    Oct 13, 2010 at 2:33
2

Short answer: The production code tests the tests.

Compare this to the credit / debit model used in economics. The mechanics are very simple - If the credit differs from the debit there is something wrong.

The same goes for unit tests - If a test fails it indicates something is wrong. It might be the production code, but it might aswell be the test code! This last part if important.

Note that your type (1) bugs cannot be found by unit tests. To avoid this types of bugs you need other tools.

Improve this answer
-1

Not an answer (I do not have the privilege to comment), but was wondering if you forgot other reasons for developing test cases...
Once you figure out all the bugs in the tests, you can regression test your application easily. Automated test suites would help you find problems earlier, before integration. The changes to requirements are relatively easier to test, as the changes can become newer, altered version of older test cases that pass, and older cases stay to pick up failures.

Improve this answer
-1

A unit test is correct if it passes correct application code and fails incorrect application code.

If your unit tests are written before you have functional application code then your unit tests should fail, otherwise they are incorrect for passing incorrect application code.

As soon as you have application code that is supposed to be working correctly, the unit test should pass. If it doesn’t pass, the application code, the unit test, or both are incorrect. If you can’t find problems with the application code then the unit tests are suspect and needs checking and possibly fixing.

The unit tests can also be incorrect because they don’t find problems. Here the writer of the application code will have an opinion what faults should be found. So starting with a passing unit tests, we can modify (temporarily) the application code to be faulty in a way that we expect a unit test to find. If the unit test fails, great, we undo the change provoking an error. If it passes then the unit test has a problem.

Improve this answer
-1

Other suggest to "write small and error-prone tests". That is imho reasonable default. I will provide two more things, which can improve quality of tests, or at least provide some metrics about their quality:

  1. consider not to write tests, but write properties (i.e. use PBT, property based testing). Just have a look at linked video and use https://jqwik.net (quite well supported/documented/developed, feature rich thing)

  2. Use mutation approach. Library, which will mutate your business class in random way and execute tests. If tests still passes, there is space to improve your tests to catch more problems perhaps.

Improve this answer
2
  • 1
    Is this answer A.I. generated?
    – Dominique
    Nov 28 at 14:06
  • 1
    Why do you think so?
    – LvargaDS
    Nov 28 at 15:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.