6

Recently I began a new project to re-implement a core part of automation. Since it is very important, I'm TDDing it so that I can tests various basic scenarios as well as things we know the old system doesn't do.

In doing this, I find myself creating a few "sandbox" tests, which exercise "candidate" algorithms in complete isolation so I can choose the one that best meets requirements. I've also created some tests to figure out why other tests are failing, which again are coded in a "sandbox" manner and don't test actual production code (they instead have a simplification of the production environment that I can alter until I find the problem/cause/solution). Neither of these has continuing value for regression; once I pick an algorithm I may never use the others, and once a test shows why another test is failing, I can correct the initial test (or its exercised production code) and verify correctness, and I no longer care that the other test's hard-coded exercise environment still fails.

When learning TDD I was taught that you NEVER delete a unit test; even if the test's original purpose has become irrelevant, the code within the test can be of use to other developers as a model, or the test can be refactored for some other purpose later. Instead, if a test has become redundant or irrelevant and should no longer be run, pass or fail, it should be ignored.

Is this the "generally-accepted" stance on ignoring tests? Does this have the potential for abuse (e.g. "I don't think it matters anymore, and it fails, so I'll just ignore it")?

Improve this question
4
  • 3
    "I was taught that you NEVER delete a unit test". Can you provide a quote, link or reference for this? It seems like the "never" is meant to stimulate thinking, not be taken literally.
    – S.Lott
    Commented Feb 13, 2012 at 21:45
  • I don't have a printed quote; this is what I was taught by senior Agile/TDD programmers at my last job, which was my first exposure to both disciplines. The reasoning was exactly as I stated; tests should pass, if they don't pass then something needs to be fixed, and if it is more feasible to fix the test by replacement then the old test becomes irrelevant, but in all these cases the old test is never deleted; it's either made to work or kept for documentation. Deleting a test means it's no longer available for use in any form, and it's suspicious; what are you trying to hide?
    – KeithS
    Commented Feb 13, 2012 at 22:40
  • 1
    A non-working test is "kept for documentation"? That's not really very helpful at all. "Deleting a test means it's no longer available for use in any form, and it's suspicious; what are you trying to hide?". That's quite a paranoid suspicion. A bad test -- one that's wrong, doesn't fit the requirements, doesn't test something useful, doesn't match the current API, etc. -- must be deleted. There's nothing to hide in removing a test that's of no value.
    – S.Lott
    Commented Feb 13, 2012 at 22:58
  • 2
    If you're using source control like SVN or GitHub, then deleted tests never die; they just get logged into history. A "redundant test reworked to make it pass" is probably more unhelpful than even a test that hasn't been written when it should have.
    – HorusKol
    Commented Feb 14, 2012 at 2:44

2 Answers 2

Reset to default
15

once I pick an algorithm I may never use the others,

False for well-written tests which exercise the interface of the algorithm.

If your tests exercise implementation details, and the implementation no longer applies, then the tests no longer apply. Delete them.

and once a test shows why another test is failing, I can correct the initial test (or its exercised production code) and verify correctness, and I no longer care that the other test's hard-coded exercise environment still fails.

Um. You should fix the "other test's hard-coded exercise environment" so that it's meaningful and useful and correct.

If a test is wrong (even if redundant) it should be corrected so that it's right.

TDD means tests are right, first and foremost. Code will get right eventually.

"and it fails, so I'll just ignore it"

This is not a potential for abuse. This is abuse.

You cannot ignore a failing test. If the test is wrong it must be fixed.

Removing tests that seem redundant is a waste of time. Extra testing never hurt anybody. Indeed, it may uncover super-subtle errors that pass one version of a test by fail another version of a superficially similar test. This may indicate design errors in an interface more than an implementation bug.

Improve this answer
5
  • Very good points. Specific example: I had a test that exercised business logic that would involve persistence. I mocked out my Repository using the mocking framework (Rhino) and the test failed. I suspected the issue was the mock itself, and wrote a test specifically exercising the mock in an equivalent "sandbox" situation. It failed; I now know my original test failed because of a flaw in the mock. I substituted a handwritten mock and the SUT then passed. The mocking framework I tried to use is still broken. I would like it to EVENTUALLY pass but currently it doesn't matter. What of the test?
    – KeithS
    Commented Feb 13, 2012 at 20:27
  • 2
    @KeithS: This scenario does not fit any of the categories in your question. You might want to update your question to include this. This is a broken test. Either fix it or replace it with one that works. If the Mock package is broken, either fix it or replace it with one that works. I'm unclear on why you need to ask what to do when your unit test toolset is broken. If you need help getting your mocks to work, then StackOverflow is the place to ask specific questions.
    – S.Lott
    Commented Feb 13, 2012 at 21:42
  • 1
    ... And I did ask such a question; it has not been answered specifically yet. This specific case was one instance of my writing a test to find out why another test is failing, from my OP. And while my specific question has gone unanswered, related questions suggest that other mocking frameworks like Moq have exactly the same problem.
    – KeithS
    Commented Feb 13, 2012 at 22:28
  • "Removing tests that seem redundant is a waste of time. Extra testing never hurt anybody." Disagree - redundant code is often a problem, even with tests. You may need to refactor them, or change them when requirements (or your test setup) change. Then the duplication will cause extra effort.
    – sleske
    Commented Dec 15, 2014 at 16:23
  • "Extra testing never hurt anybody" - Nope. If your tests start taking too long to run, the devs run them less. Truly redundant tests increase test time with no benefit and should be removed. If you're not sure it's redundant, leave it, but if it truly is redundant get it out of there.
    – Michael Kohne
    Commented Feb 8, 2016 at 13:06
4

I would rather have all of my tests pass at any given point when they are executed. If a test fails and there is no documentation explaining why it is expected to fail, then it shouldn't simply be ignored. Any other developer working with these tests will be confused since they will not immediately know whether a recent change they made affected the outcome of any failed tests.

Imagine having to explain to a co-worker or manager why certain failed tests are to be "ignored". I don't think you will have an easy time explaining to them why these failures are a positive thing and do not need to be addressed.

Remember that you are testing your code to ensure correctness. Failures should only ever indicate that the code being tested is incorrect, and incorrectness should never be ignored.

Improve this answer
2
  • 2
    Good point. You can in most unit-testing frameworks (I'm using NUnit) document a reason for ignoring the test. That reason should never be "because it's failing". Reasons like "tests obsolete code" or "does not test production code" are valid IF truthful, IMO. My question was basically if that was the consensus; I infer so far that you do not agree.
    – KeithS
    Commented Feb 13, 2012 at 20:17
  • 1
    As @S.Lott mentioned, if it tests obsolete code I would simply delete the test and the obsolete code as well. I would never leave a test that can be executed and that will always fail. It should sound alarms and initiate action to fix the test so that it passes.
    – Bernard
    Commented Feb 13, 2012 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.