5

I'm really new to TDD, so I guess this question is pretty basic.

We're building a website, and part of the functionality is generating some files (binary files: Excel, PDF, whatever). How should I test this feature?

I've think about creating some static files and compare those with the generated ones, but a binary comparison isn't reliable (the files can have the same content but different checksums), and, if I understood TDD correctly, a logic comparison isn't a good idea, since I should use basically the same algorithm I use to generate the files, so I wouldn't be really testing anything.

How are this kind of things usually dealt with?

  • "since I should use basically the same algorithm I use to generate the files" generally I don't think this would be true. – Ben Aaronson Oct 6 '14 at 23:21
  • I mean I should use the same library I use to generate the file, not the same data algorithm. Not sure if it's the way to go, sounds like too much coupling. – César García Tapia Oct 6 '14 at 23:24
4

I've done this by using a library to parse the files. Sounds like you already have one available so you should use this.

Even if you could compare the entirety of the binary files, I wouldn't recommend this. Lets say you've got 100 tests that are testing various things by comparing the entire binary output. Then a new requirement comes in that the title of each file should be changed from "Foo" to "Bar". Assuming all your output files were outputting a title of "Foo", you've now got 100 broken tests you have to fix.

A better way is to only test one thing in a test that doesn't overlap with the other tests. In this context, only a few tests should be responsible for checking the title. Then if the same requirement comes in, only those few tests would need to be changed.

  • Parsing the file is in the lead of possible solutions, but I wonder if that's placing too much logic in the tests. If I write a parser to test the files, I guess the parser should be tested too. Sounds painful in the long term. – César García Tapia Oct 7 '14 at 7:57
  • @CésarGarcíaTapia I thought you have a library to parse the files already? I was writing my answer assuming that. There are definitely java libraries for parsing Excel files and probably PDF. – Daniel Kaplan Oct 7 '14 at 16:19
10

How are this kind of things usually dealt with?

By not unit testing the files.

Generally, there's some intermediary format that defines the content of the file. You would test that the intermediary content you expect is being sent off to be PDF-ified. Basically, isolate everything but actually encoding your output to PDF. In an ideal world, that work will be done by some library, so you don't need to test it.

In a realistic world, you need humans to eyeball the PDFs to make sure that they're well formed and "look right" anyways (and load properly in Acrobat on different platforms, etc.), so have them check the content too. Not everything is a good fit for TDD.

  • That's another good alternative. Technically you could still be practicing TDD without doing this though as TDD doesn't have to be done with unit tests (not that you said otherwise). But, if this gives you enough confidence in your code, it'd be preferred. – Daniel Kaplan Oct 7 '14 at 0:10
  • Essentially, you split your code into your business logic, and the interface logic to drive the PDF generator. The business logic is then unit-tested. The interface logic should be so simple you don't need unit tests - of course it will still need integration tests. Manually checking the files would be one kind of integration test. – sleske Oct 8 '14 at 9:53
  • 1
    Once you separate the business logic from the PDF generation, you could use a file checksum to make sure the PDF library is outputting byte-perfect results. – raptortech97 Oct 8 '14 at 22:47
2

Testing output files is always a difficult thing, same goes for testing downloading files from the web or output to the console.

One question you should ask yourself is: "How far can I test until I need the file?" Most of the logic can be tested by using some kind of replacement code, or simple text file generator.
Looking at your question you can generate multiple files, so I guess you already have some sort of separation between the code that provides the data and the code that is generating the file. So my answer will be based on the assumption that there is a kind of factory that generates the actual file.

A question you can ask yourself is: how complex is the factory, does it contains lots of extra logic or is only calling functions from another library that is creating the files for you?
If the factory does have lots of logic, is this the same for all kind of outputs or different for each output? When it is the same, you should think of refactoring it out of the factory, or add a format that is easy to test (like a plain text file).
When it only calls the a library, do you really have to test it automatically? aren't you going to test the library you are using?

So basically I would test as much as possible that doesn't require me to create a complex file. Plain text files, including html and json are relative easy to test because they are readable and comparable. To test the binary outputs, I would probably generate a simple sample file to test against and write a test that uses the factory to generate the same file and test the file at byte level. Generated with the same parameters should generate the same file over and over again.

2

File output usually belongs to integration-tests(=having some components working together) and not to unittests(=test one component in isolation)

if your pdf-generation is implemented in a way that the same input always produces the same output you can try approvaltests which does a binary compare to the previous call result. if there is no previous result or the binary compare is different a gui asks you if old and new version are the same showing you both pdf files with acrobat reader.

This way you are informed every time the output changes which might be ok or not. Example

note:

if the pdf-generation inserts the current date into the output then the ouput will be different on every call.

If you provide the date as a parameter to the pdf-api then you can generate identical output.

  • ApprovalTests has the same issue I mentioned in my answer (see "100 broken tests"). Although it is a good tool to be aware of, I think it's better for building a safety net around something that's difficult to test rather than an end result. – Daniel Kaplan Oct 7 '14 at 16:24
1

There are a few different ways of testing files:

  1. Binary comparison. Output a "golden" result file for each test; manually confirm it's right; store it and compare it against future file outputs. Pros: Simple. Cons: Fragile. Prone to false negatives (esp if files contain metadata, like 'creation date' that change even when the 'payload' data does not.)
  2. Parsing the file. Use a library to parse the (Excel, PDF, text, ...) file. Run assertions on the parsed data. Pro: Less fragile than binary comparison. Can easily avoid metadata flutter. Cons: More difficult and complex to code. Not all file formats are readily parsed, nor are the interesting features of the output amenable to a concise, descriptive set of test assertions.
  3. Rendered comparison. Output the file; use a rendering engine to convert the file to a more comparable representation (such as a lossless image file). Pro: Can be less fragile than pure binary comparison, and easier to code than parsing plus assertions. Focuses easily on 'payload' rather than metadata. Cons: Dependent on renderer and rendering envionrment (including specific renderer version). Requires same "golden image" files as straightforward binary comparison. May require cropping, filtering, and sorting operations to align and compare only the relevant parts.
  • Good points. However, I would class all these tests as "integration tests", because they involve multiple components, and even external tools/libraries. – sleske Oct 8 '14 at 9:57
  • @sleske Perhaps. I do not abide traditional "this is a unit test, that is something else" boundaries. Creating output files involves something external (the OS/filesystem), so cannot be (meaningfully) tested without reference to that external context. I'd argue that file output cannot be well nor easily tested in classic unit tests that don't involve some external components. To me, that's an indictment of unit testing as the focus of developer attention. I'd rather use more things and get a robust test than fewer things and a cleaner but weaker test. YMMV. – Jonathan Eunice Oct 8 '14 at 15:06
  • I don't want to get dogmatic - I just find the distinction between unit and integration tests helpful, because both have different strenghts and weaknesses. And yes, some things are not pratical to unit-test - that's why we have integration tests :-). – sleske Oct 8 '14 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.