3

I have recently become responsible for a legacy tool that analyses code and provides a log as output. As part of the JUnit suite for this, there are ~100 tests that rely on successfully matching the produced log with a static expected log.

Now this means that if the format of the log changes at all, it will break the unit tests. But I cannot decide if this is a good or a bad thing.


Pros:

  • Quickly failing unit tests let me know that something has changed

Cons:

  • Tests have to be updated every time the log style is changed
  • A straight text compare is bad for this case, because the log file involves times. So if a run doesn't take exactly as long as was expected, the test fails.

I can't help but feel that these tests shouldn't be run. But at the same time I don't want to leave the log file process uncovered.

What is the solution here? Better unit tests? No unit tests? How can I do this without my tests being so brittle?

The log is being written to a file, and is not part of any logging framework (a file is generated after the tool runs, telling the user which files failed validation and why).

  • 1
    Are you validating the log file contains some string? How are you logging to the log file? Could you instead test the "this object to string" methods? Can you change the appender associated with the log framework (you are using a logging framework... right?)? – user40980 Jul 17 '15 at 13:26
  • The tests are validating a log file that is output by the tool. This log file is compared to an expected log file using FileUtils. If the files do not exactly match then the test fails. (Note I am fairly sure that whitespace is being taken into account with these comparisons) – Rossiar Jul 17 '15 at 13:59
4

Short-term solution: Leave things as they are. Badly written, brittle tests are much, much better than not having tests. On a scale from "no tests" to "tests so reliable they make you weep and do all your work for you", your existing test suite is much closer to "weep" than to "nothing".

Middle-term solution: refactor. It's inconceivable that the analysis tool doesn't have an intermediate representation of its findings, before it's turned into a stream of text. Make that representation publicly accessible and rewrite the tests to assert things about that representation, not about the textual form. It sucks, but you only have to do it once, and from then on maintenance will be much, much easier, even when logic changes and new cases are added.

  • I am looking again at a different piece of code and it doesn't seem right to put methods in that make code publicly accessible just for testing, is this bad design? – Rossiar Aug 14 '15 at 10:49
  • 1
    @Rossiar NO! It's not bad design to make things accessible "just for testing". Your test suite is a central part of the code base. The fact that it is never shipped to the customer is irrelevant - everything that consistently improves the quality of your product is by definition good, and making things testable that otherwise would be hard or impossible to test satisfies that definition. – Kilian Foth Aug 15 '15 at 18:02
  • Thanks for the reply. If that is the case, then why have private methods? Surely there are security concerns to just opening up code for unit testing purposes? – Rossiar Aug 17 '15 at 9:41
1

If the format of the log changes at all, it will break the unit tests

When code changes, it breaks the corresponding unit tests. By following your logic to the extreme, you would avoid writing tests at all.

Think about the goal of those tests. Are you testing:

  • That a method actually logs something?

  • That a method uses a proper message template?

  • That a method uses proper values to format the message?

  • That a method logs a specific message (that is a message template formatted using the actual values)?

In the first three cases, you may use dependency injection (DI) to inject a logging mock or stub. On the other hand, this means more code and more complicated logic, so if you are pretty sure that log format won't change anyway in the near future, this may not be a pragmatic thing to do. Note that most logging frameworks make it very simple to substitute your own handlers during testing, which means that the actual approach may be much simpler than implementing DI.

If you're actually testing that the messages correspond to the expected ones, then the current solution responds perfectly well to the requirements. Those hundred tests are not unit tests, but functional/system tests; still, they are perfectly valid in their current form.

Consider logging as one of the outputs of the method which should be tested. Like you assert that the actual result of a method is equal to the expected value, you do in the same way the assertions about the logs. Ideally, in unit tests, the testing surface will be very small: you'll inject a logging mock and do an Assert.Equals on the messages received by the mock, in the same way you'll be testing that a method which should export data to a file actually did its job without having to actually use the file system. In functional and system tests, the testing surface may be much larger, and include the actual file system.

1

The underlying problem that you are encountering is that you are testing everything in one go when comparing two text files. If any little thing changes, then the entire test fails and it is indeed tedious to find the part that broke and then fix it.

You are doing system testing in jUnit. Not that it can't do it (though there are better testing frameworks for incorporating multiple types of testing) but it is important to realize the distinction between the types of tests and what you are doing.

And yes, system tests are some of the more fragile tests that one can do.

The first step to correcting this would be instead of writing out a log file yourself, use a logging framework such as Log4J and have a custom appender write out the format you want to the final file.

By switching to a logging framework, you are then able to use different appenders for different situations. When running in test, you can have your own validation appender. This would allow you to expose the information that is sent into the logger in the tests and verify that the information is correct - that when you send this message into the logging framework, it contains this information. That is a unit test.

Another unit test would be to validate that when you send specific information into the 'to customer file' appender then the corresponding information is properly formatted in the log (and you mock the log being written so that you are not going out to hit the disk with the comparison). Each type of message is tested and the format validated. This allows you to test the format in isolation from the complete product. If the format changes, you fix it here (rather than in all the other tests).

You may wish to look at testNG which has as part of its design goals the ability to separate test suites and test phases (unit, integration, system) better than jUnit did at the time that testNG was written (jUnit has come a ways since then). This would help you make the unit tests easier to run for developers (which makes them more likely to be run).

  • This is pretty close to the answer I would have written. I would add that a quick and dirty way to validate the log that actually might work well is rather than comparing the whole log, pattern match it. Most of the contents are likely irrelevant from the perspective of a unit test. Make sure it contains that one phrase or sentence that indicates the meat of the result. I have done this before with good success, but it is not always appropriate so one must use caution when coming up with a strategy to validate something like this. – user22815 Jul 18 '15 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.