Should we always unit test bugs when correcting them?

Question

When correcting bugs, it is encouraged where I work to first write a test that fails with the given bug, and then to fix the code until the test passes. This follows TDD practices, and is supposed to be good practice, but I noticed it tends to produce cryptic tests that come really close to the implementation.

For instance, we had a problem when a job was sent, reached a certain state, was aborted, and retried. To reproduce this bug, a massive test was written with thread synchronization in it, lots of mocking and stuff... It did the job, but now that I am refactoring the code, I find it very tempting to just remove this mammoth, since it would really require a lot of work (again) to fit the new design. And it's just testing one small feature in a single specific case.

Hence my question : how do you test for bugs that are tricky to reproduce ? How do you avoid creating things that test the implementation, and hurt refactoring and readability ?

Answer 1

Yes, in general you should. As with all guidelines, you'll need to use your best judgement when they run against other guidelines. For this scenario, the severity of the bug needs to be weighed against the work needed to implement the test and the quality of that test in being targeted to the business problem and catching regression of the bug's state.

I would tend to favor writing tests over not, since interruptions to run down bugs tend to have more overhead than simply developing and maintaining a unit test.

Answer 2

I think the best practice - one I'm embarrassed to admit I don't often follow - is to create a system or integration test that demonstrates the problem observed in production, then research to find the unit(s) responsible for the problem, and then write unit tests for those units that demonstrate the problem at the unit level. Once you have the unit tests, fix the units, and run all the tests. At this point, it may be prudent to discard the original test - because it may be fragile and/or slow - but keep the unit tests in your automated suite for the sake of regression and coverage.

Answer 3

The practice of writing a test to identify the defect is a good idea, since it allows you to identify exactly what steps are needed to reproduce the defect and verify that it has been fixed. In addition, these tests can be run as part of smoke testing or regression testing to ensure that later changes have not re-introduced an old defect into the system.

The first thing to consider is the level of the test needed. Perhaps the test to verify the fix would be more appropriate at a systems level, or maybe even an acceptance test that's done manually. I think it's more important to have a test that's documented and managed, regardless of how it's specifically implemented.

As far as how refactoring affects tests, it depends on the specific characteristics. In some cases, refactoring (or any type of work, such as new features) could make the tests no longer necessary. The problem, as it originally occurred, might no longer be possible. In that case, it might be wise to remove the test from the possible tests to make your test process (automated or manual) more lean. In other cases, there are multiple methods of performing the test and verifying the feature at a different level might be more appropriate. If the feature is minor, perhaps the test is no longer necessary at all.

You can also consider not only relying on testing, but logging as well. For example, capturing information at run-time (with varying levels of verbosity depending on the environment - more verbose during testing, less verbose during deployment), profiling the application, capturing dumps of the current state of the system. If you can find common triggers to the problem, you can use that to guide testing at all levels.

Answer 4

Yes, you should.

Write unit tests for existing code base. When fixing a bug, you need to make sure that your unit test fails - this will give you confidence that you are indeed working on a bug. You then need to re-factor and make the test pass by fixing the bug.

This is not a TDD practice though. In TDD tests drive your design, but in your case the design has already been decided upon.