Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
I am running unit tests written by someone else, and the output is littered with print statements. I know libraries are not supposed to contain print statements, but is it bad practice for tests to contain them? My take is that they shouldn't, and asserts and other comparison statements should be employed.
Not sure that this is relevant, but the codebase is in Go.
Let's take a step back. We use printing / logging so we can diagnose what is going on under a particular code flow, such that we can take a look back at wherever these logs are being aggregated to see what happened at a past time.
We use tests to assert that a particular code flow operates as expected. This can be for an isolated part of the code flow (a unit test) or a broader part of the code flow encompassing multiple moving parts (an integration test). Having assertions within these tests is like logging in that it generates output about a particular code flow. I'd say that we can compare this to structured logging, since these outputs are standardized and can be parsed, along with the test processes having meaningful error codes. For example. unittest-xml-reporting parses unittest.
This all contributes to these types of tests being automated, their protocol is standardized. My question for you is, what do you want to accomplish by having print statements in your tests?
If it's for a sanity check, realize that this can be made into a formal assertion instead. Also consider that these tests are usually automatically invoked (such as from a build server), and often if they run successfully their outputs won't be dived into. Attempting to parse a print statement if you do wish to automate this sanity check should not be favored over formal assertion on whatever the print statement was printing in the first place.
If it's to say what a test is doing, this comes down to naming tests properly instead of printing some description during the test's execution.
I think these two intents cover what a print statement would potentially do, so you should consider what scenario you have and adapt accordingly.
I’ve been writing debugging code since before unit tests were a thing. Only form of commented out code I ever tolerate. Over all that time one thing remains unchanged: you need to be able to easily turn it off.
Use comments, debug flags, structured logging, whatever. But never force anyone to look at this noise when they don’t care.
When I’m focused on getting a test to pass detailed info might be handy. When I’m thinking about another test please give me an easy way to turn the noise off.
Fail to do that and I’ll remove the print statements without mercy.
For automated tests, it is essential that we can easily determine whether a test succeeded. Typically this means: either the test program exited with status 0 (success) or a non-zero exit code (failure). It should not be necessary to manually analyze the output to determine whether the test succeeded.
This means that printing debug info or further details about test cases is generally fine. Some test runners can also buffer such output and will only show it upon failure.
There are also richer, more structured test reporting formats than just the exit code + unformatted debugging output. A very simple format is TAP (test anything protocol). Here, test status is indicated by lines that start with ok or not ok, and other output lines should start with a # comment. The JUnit XML format is also pretty common for test reporting. Such reporting formats are very useful if an executable contains multiple test cases, and we would like to know which of the many tests failed.
Start from the beginning. Why do you use unit tests (apart from it being fashionable or required at your company):
In order to gain some confidence that your application works as intended, because you know that many units work as intended.
In order to be able to make code changes and be confident that you are not introducing stupid bugs because your unit tests continue to work, or to be notified that you introduced some stupid bug because unit tests fail.
In order to have some help that leads you to a bug fix quicker if a unit test shows a bug.
How do these print statements affect this? It depends on your development environment. Print statements that interfere with your unit tests (for example if the output is parsed by some tool) are to be absolutely avoided obviously. Print statements within successful unit tests should be avoided. Print statements in failed unit tests: You decide whether they help you with (3). Do they help you fixing bugs quicker? Or not? If they help you, leave them in. If they don't help, remove them. But there is no fixed principle "no print statements in unit tests".
assumption: you already use a testing framwork and an assertion library that enable you to know about when a test is executed and which is its outcome
I'm not that kind of gut who blindly believes in best practices. To me, everything is feasible and everything is good unless someone shows me a concrete downside. what are downsides of logging test code? performance? readability? if not, go with it
There is somethig, though, that we have to consider: in general, the logging purpose is to observe interal state of the system at work. Logging helps to investigate about misbehaviours beacuse lets to recognize problems with inputs, outputs, exceptions and with components integration
Unit tests are supposed to do the exact inverse: they are black box, they are "all of nothing", executed in isolation and without tolerating exceptions, if not part of side effects checking.
So, adding logs to tests seems a real code smell. Why you need them? Are your tests long and complicated? you are not able to investigate on failing tests? production code logging entries are not enough to explain the flow? This is something to discuss about
