Stronger Unit Tests Unearth Many Bugs, What to do re Validations?

Question

I recently upgraded some tests that unearthed a lot of previously hidden bugs. The bugs are rare and low priority, but they are still bugs and will eventually need to be fixed.

How should I handle the changes to the tests?

If I check-in my changes, many existing validations will fail, and given that these bugs are relatively low priority, they won't be fixed any time soon. The validations will have to be turned off until they get fixed. However, if we turn off so many validations, a larger regression may go unnoticed.

Another option is to check-in my changes but use error messages instead of validation failures. This clutters up the logs, and these bugs may go forgotten.

What are the best practices here?

Answer 1

Am I stating the obvious?

The best practice, here, is: "Talk to Your Manager."

This check-in will have repercussions and you know it, it could scramble the workflow, people could decide to fix those bugs, and miss deadlines on features they were working on, maybe some of those bugs are really straightforward to fix, but some people's work will be diverted around this event.

There is some managing to do.

Also there may be parts of the bigger picture that are obscured from your sight and consequences you cannot foresee. (interdependencies/deadlines/contracts/versions and libraries that are already deployed in production)

If I was you manager I would appreciate your consideration before checking this in and would advise you on how to better handle it.

If you are the manager, I wonder why you're even asking.

Answer 2

nUnit has [Ignore] annotation. I would guess other testing frameworks have similar way to ignore test cases.

Any test runner then shows how many tests were ignored. So you can track how many tests are ignored and build some kind of "fix all/some ignored tests every week/sprint" policy.

Answer 3

The best practice is not to change the tests unless you also plan to change the associated code for any bugs you find. If you were given the assignment and priority to change the tests, then spending a day or two fixing unearthed low priority bugs is, as Anakin would say, implied in your mandate.

If you've just inadvertently uncovered weeks of low-priority work, which is doubtful, then you need to put user stories on your backlog or issue tracker so the work isn't forgotten. Most test frameworks let you mark a test as ignored, so it is still present, but gets reported and counted as ignored so it doesn't get forgotten completely, and must at least be maintained to the point of compilability.

Answer 4

The bugs may be low priority but the brittle tests that fail because of these changes are a menace. Fix the tests now. Tests are meant to make refactoring easier not harder. Be careful not to simply write the next generation of brittle tests. Once they're fixed reevaluate how you feel about your "low priority" bugs.

Test's are not a silver bullet. You don't magically gain superior wisdom because you've written a test that seems to make sense. If the bugs you uncovered had obvious and easy fixes you wouldn't be asking you'd just fix them. You want it to be easy but it's not. Now you've got a failing test and you don't know what to do with it. I know three solutions here:

• Fix the problem
• Discard the test
• Write a better test that makes the problem easier to fix

You've said something that I'm still waiting to make sense. Why would validations that rarely fail need to be turned off? Validation is supposed to catch bad input before it becomes state. That happening rarely isn't a reason to prefer rare bad state over rare validation errors.

I've no idea what problems you're referring to but the only thing I hate more than things that fail quietly are things that fail quietly only when they feel like it.

The only good reason I can think that this is tying you in knots is this is happening someplace that you don't have any good way to handle a validation error. If that's the problem you need to create one.

Answer 5

The rule should be that all tests should always succeed, and if they don't pass, bugs need to be fixed (taking into account that a unit test may have a bug and need fixing).

That said, if your team would be in trouble if you commit all your unit tests right now, they could be put on some branch, and whenever someone has time to work on it, they pick a few tests, run them, fix the bugs, and commit those tests with their fixes.

Answer 6

MSTest offers a TestCategory attribute. That is, you could mark those tests with

[TestMethod]
[TestCategory("WaitingBugs")]

During a "normal" build, you can exclude tests by category,

mstest.exe ... /category:"^!WaitingBugs"

True, one issue remains: someone needs to remember that there are some tests which are normally not run...