I have recently become responsible for a legacy tool that analyses code and provides a log as output. As part of the JUnit suite for this, there are ~100 tests that rely on successfully matching the produced log with a static expected log
.
Now this means that if the format of the log changes at all, it will break the unit tests. But I cannot decide if this is a good or a bad thing.
Pros:
- Quickly failing unit tests let me know that something has changed
Cons:
- Tests have to be updated every time the log style is changed
- A straight text compare is bad for this case, because the log file involves times. So if a run doesn't take exactly as long as was expected, the test fails.
I can't help but feel that these tests shouldn't be run. But at the same time I don't want to leave the log file process uncovered.
What is the solution here? Better unit tests? No unit tests? How can I do this without my tests being so brittle?
The log is being written to a file, and is not part of any logging framework (a file is generated after the tool runs, telling the user which files failed validation and why).
expected
log file using FileUtils. If the files do not exactly match then the test fails. (Note I am fairly sure that whitespace is being taken into account with these comparisons)