Should log statements be tested?

Question

When writing tests I usually ignored logging statements, but now I wonder if it was right.

The logs are often important tools when diagnosing production issues, moreover there can even be requirements for logs, like "log every interaction with external system, including request and response on highest detail level" or "never ever log unmasked sensitive data".

From my experience such requirements were tested by the functional testers sifting through pile of logs with some scripts. But maybe it would be better to ensure the logging is correct closer to code - with unit tests.

Now the question is should the logs be tested with unit tests?

I'm asking because I feel that often unit testing logging would require considerable effort because

• Often involves parsing stream of strings
• Capturing the logs in test case in may be not straightforward
• Capturing logs in a way that that ensures they ar not interlaced with logs from other sources may be even less straightforward

On the other hand I believe quality of logging in application is important and unit testing may be a way to ensure that quality, but maybe other ways are more cost effective.

Answer 1

Yes you should absolutely test your logging. I said logging and not logs because they don't matter much, as logs are a detail of implementation.

What I mean is you should test that the action of logging was done. Either by using a mock, a fake or anything you'd like. You'll find it a lot easier to test that behavior, and your code will also be cleaner as it'll depend on the concept of logging, not the log files concretely. That means you'll be able to change your strategy of logging as a whole depending on anything you'd like (hint: testing environment, cough, fake logging in memory, cough, easy assertions).

In case you come to think your logging is not tested "for real" (and you will), understand that only the implementing class(es) of that logging concept should be tested "for real". Not classes which use the logger.

So now classes using the fake logger prove they're using it the way they should, and implementations of the logger prove they work.

Answer 2

No.

Logging is a code aspect, not required behaviour: Your application would behaviourally work just as well without the logging.

Quoting from Wikipedia: Aspect (computer programming) :

An aspect crosscuts the program's core concerns, therefore violating its separation of concerns that tries to encapsulate unrelated functions. For example, logging code can crosscut many modules, yet the aspect of logging should be separate from the functional concerns of the module it cross-cuts. Isolating such aspects as logging and persistence from business logic is the aim of aspect-oriented software development (AOSD), of which the aspect-oriented programming (AOP) paradigm is the most widely employed.

Unit tests are for testing business logic. If it's not business logic, don't test it in a unit test.

If you have utility methods that assist logging, you should unit test those.

If you want to "test" logging, do it as part of integration testing and assert that some stream (usually console) contains certain output.

Don't be too prescriptive over the form of the log message: Use regex or similar so assert the log message "looks OK" (has at least certain info in it), that way you can tune the log message for readability without breaking the test. "Looks OK" can be prescriptive about format though, especially where log aggregation and searching thereof requires it.

Answer 3

If logging is an expected behavior of your system under test I would definitely suggest writing tests for it. I don’t see a reason why you shouldn’t.

Answer 4

yes, you should test certain aspects of your logging, especially if you operate system that needs to meet certain traceability requirements (who did what when).

the problems you state are valid. One way to make life bit easier is to use "structured log" producers — fluent API or other way to pass parameters to log line template or to build log lines. Such structured loggers are easier to test, mock and stub.

Answer 5

I do agree with Steve that you should test your logging interfaces to make sure they behave correctly, but if I understood your question, you're more focused with testing the log output itself.

I think that depends on use cases but in my use case and domain, what the logs capture is actually a "last-resort option" for issues that escape testing. If some bug flies under the radar and a user crashes somewhere, for example, their log shows us what the system was doing when they crashed which significantly narrows the list of suspects (unless, of course, the log output is wrong/misleading which is probably what you're trying to avoid, but that seems so out of the way to me to test).

So it seems rather redundant in my case to test the output of the logs themselves, since in an ideal world we wouldn't need them. They're a last resort measure, and have been a lifesaver in some contexts (one time I was able to quickly narrow down that a user's hardware didn't have support for SSE 4, even though we listed it in our requirements, by looking at his log combined with hardware requirements; absent the logs I might have spent a week bouncing builds back and forth with him to try to narrow down where he crashed). And we solved that by at least making the code detect that hardware limitation and report it to the user instead of crashing*

Though we had to bear the bad news that he did not pay attention to our minimum hardware requirements; he was using some obscure prototype machine we never heard of which still didn't support SSE 4 in 2013 in spite of having 24 cores and 64 gigs of DRAM. Later, out of sympathy, I actually spent a weekend porting the code to use SSE 2 in those cases to reduce our minimum requirements since I figured he must have invested enormous sums of money for that prototype hardware even though there wasn't a legit business requirement. It made me sad to think a person with such beefy hardware couldn't run our software because of this restriction.

But in an ideal world I wouldn't lean on the logs for such ad-hoc debugging; all such issues would be caught in our tests. But I can't always depend on our tests for that, especially with varying hardware capabilities (when our team, including QA, doesn't have all the hardware in the world to test against, even though they have a wide range). Yet testing the log output would be quite a time sink with probably little gain for an issue that ideally should have been caught absent the logs in the first place.

For my domain (and I don't expect everyone to be the same here), we treat logging like a "non-side effect". That is to say, we don't unit test functions to make sure their implementations write the "correct" things to logs as part of the functional requirements, because that would double up our testing efforts for something designed to catch what eludes our tests in the first place. Even "correct" logging isn't that interesting in our case, provided it's not redundant. If some subsystem writes, "I'm doing backflips and eating pizza!" which doesn't describe, very well, what they're actually doing, as long as no other systems are writing that same info, it still lets us trace down exactly what the software was doing before it crashed or glitched out.

We do, however, have tests designed to make sure the logging functionality itself works, and across threads, and against varying exceptions. But that's separate from testing the logging output of every single thing that utilizes logging.

Answer 6

You should test two things: 1. Does logging work? If I put a logging statement in the source code, will there be logging output where I want it to be? And will it show what it is supposed to show? That is tested just like any other functionality in your code.

But also you should check that your logging statements are useful. If users complain about problems, will your logging statements help you find the reason for the fault and tell you how to fix it? Will there be tons of logging output that doesn’t help you? Worst, are there logging statements that seem to hint at errors that don’t exist and waste your time? That can only be checked manually, not through any automated testing.