Unit testing text output

Question

I have recently become responsible for a legacy tool that analyses code and provides a log as output. As part of the JUnit suite for this, there are ~100 tests that rely on successfully matching the produced log with a static expected log.

Now this means that if the format of the log changes at all, it will break the unit tests. But I cannot decide if this is a good or a bad thing.

---

Pros:

• Quickly failing unit tests let me know that something has changed

Cons:

• Tests have to be updated every time the log style is changed
• A straight text compare is bad for this case, because the log file involves times. So if a run doesn't take exactly as long as was expected, the test fails.

---

I can't help but feel that these tests shouldn't be run. But at the same time I don't want to leave the log file process uncovered.

What is the solution here? Better unit tests? No unit tests? How can I do this without my tests being so brittle?

The log is being written to a file, and is not part of any logging framework (a file is generated after the tool runs, telling the user which files failed validation and why).

Answer 1

Short-term solution: Leave things as they are. Badly written, brittle tests are much, much better than not having tests. On a scale from "no tests" to "tests so reliable they make you weep and do all your work for you", your existing test suite is much closer to "weep" than to "nothing".

Middle-term solution: refactor. It's inconceivable that the analysis tool doesn't have an intermediate representation of its findings, before it's turned into a stream of text. Make that representation publicly accessible and rewrite the tests to assert things about that representation, not about the textual form. It sucks, but you only have to do it once, and from then on maintenance will be much, much easier, even when logic changes and new cases are added.

Answer 2

If the format of the log changes at all, it will break the unit tests

When code changes, it breaks the corresponding unit tests. By following your logic to the extreme, you would avoid writing tests at all.

Think about the goal of those tests. Are you testing:

• That a method actually logs something?

• That a method uses a proper message template?

• That a method uses proper values to format the message?

• That a method logs a specific message (that is a message template formatted using the actual values)?

In the first three cases, you may use dependency injection (DI) to inject a logging mock or stub. On the other hand, this means more code and more complicated logic, so if you are pretty sure that log format won't change anyway in the near future, this may not be a pragmatic thing to do. Note that most logging frameworks make it very simple to substitute your own handlers during testing, which means that the actual approach may be much simpler than implementing DI.

If you're actually testing that the messages correspond to the expected ones, then the current solution responds perfectly well to the requirements. Those hundred tests are not unit tests, but functional/system tests; still, they are perfectly valid in their current form.

Consider logging as one of the outputs of the method which should be tested. Like you assert that the actual result of a method is equal to the expected value, you do in the same way the assertions about the logs. Ideally, in unit tests, the testing surface will be very small: you'll inject a logging mock and do an Assert.Equals on the messages received by the mock, in the same way you'll be testing that a method which should export data to a file actually did its job without having to actually use the file system. In functional and system tests, the testing surface may be much larger, and include the actual file system.

Answer 3

The underlying problem that you are encountering is that you are testing everything in one go when comparing two text files. If any little thing changes, then the entire test fails and it is indeed tedious to find the part that broke and then fix it.

You are doing system testing in jUnit. Not that it can't do it (though there are better testing frameworks for incorporating multiple types of testing) but it is important to realize the distinction between the types of tests and what you are doing.

And yes, system tests are some of the more fragile tests that one can do.

The first step to correcting this would be instead of writing out a log file yourself, use a logging framework such as Log4J and have a custom appender write out the format you want to the final file.

By switching to a logging framework, you are then able to use different appenders for different situations. When running in test, you can have your own validation appender. This would allow you to expose the information that is sent into the logger in the tests and verify that the information is correct - that when you send this message into the logging framework, it contains this information. That is a unit test.

Another unit test would be to validate that when you send specific information into the 'to customer file' appender then the corresponding information is properly formatted in the log (and you mock the log being written so that you are not going out to hit the disk with the comparison). Each type of message is tested and the format validated. This allows you to test the format in isolation from the complete product. If the format changes, you fix it here (rather than in all the other tests).

You may wish to look at testNG which has as part of its design goals the ability to separate test suites and test phases (unit, integration, system) better than jUnit did at the time that testNG was written (jUnit has come a ways since then). This would help you make the unit tests easier to run for developers (which makes them more likely to be run).