40

I'm writing some test code for a feature which processes PDF files. The basic idea behind the tests is that I point them towards some PDFs I've selected specially, they process them and I check that the output is what I expect.

My question is: where should I be storing these large-ish PDFs? Should I check them into version control along with the code? Or put them somewhere else? Obviously, the test code is useless without the PDFs (or even with different PDFs) but still, putting them into our repository feels wrong.

  • 2
    possible duplicate of Should unit tests be stored in the repository? – a CVn Oct 2 '14 at 20:30
  • 19
    @MichaelKjörling: Tests != Test Data – Robert Harvey Oct 2 '14 at 20:42
  • 4
    @RobertHarvey True, but if the test data is required for the test to work, I feel it should be considered a part of the test. That is also the approach taken by all three answers so far, as I understand them. – a CVn Oct 4 '14 at 13:38
84

Your version control system should contain everything it needs to build, compile, test, and package an application for distribution (e.g. MSI, RPM). I would also argue build configurations and other scripts should also be in version control.

I should be able to check out a project and have a complete compile, build, and test environment.

There are two approaches to checking in test data. First, you can check in the test data itself (PDFs in this case). Second, you can check in source data that can be used to generate test data (if applicable). This could be a SQL script loaded into a blank database containing test data, or maybe a text-based file that can be compiled into a PDF or other file.

Others may disagree with checking everything into version control, but I have found in my professional experience it is critical to ensuring a complete environment is able to be rebuilt from scratch.

  • 20
    Yes. Absolutely yes. It's 2014, there is no justification whatsoever for using revision control that doesn't handle binary files seamlessly. – Kilian Foth Oct 2 '14 at 17:38
  • 4
    I agree, but you definitely want to avoid the situation where you are checking in junk items as well. For example, if the test data includes an "output" folder that contains all the pdf files generated by the tests, then you will want to not include that into repository. But I do agree the tests themselves should be part of the repo as well as any packages needed to run it. – Kenneth Garza Oct 2 '14 at 19:37
  • 1
    @KennethGarza It isn't hard, really. As a rule of thumb, any original content (source code, test source code, test data, media, [real] documentation, third party libraries, build scripts, tooling scripts, conversion scripts, etc.) should be included, while all data that can be generated in reasonable time from the original data should not be. Besides, given those are the test outputs, they probably only make sense after running the tests yourself, otherwise you are not testing your program, you are testing the VCS software's ability to preserve the integrity of your files :) – Thomas Oct 2 '14 at 20:09
  • 1
    @MarnenLaibow-Koser: a project I worked on to detect electrical failure in implanted pacemaker leads had a test suite of over 40GB. There isn't a VCS in existence where dealing with that isn't obnoxious. Having two repos is an administration hassle of its own, but it sometimes can be the better choice. – whatsisname May 18 '18 at 2:40
  • 1
    @MarnenLaibow-Koser you got it. Integration tests are in separate repo and if user wants to run it locally, dependency management will fetch zip file for him and decompress it. Usually Continuous Integration server/farm is tasked to do integration test and will prevent merge feature branch until integration tests pass. – user482745 Nov 22 '18 at 13:59
15

If the tests are useless without the setup files that you have prepared, then it makes sense to include the files in your VCS along with the test code.

While the files used in the test aren't code, you can view them as a dependency that the code relies upon. So there is merit in keeping everything together.


As a counterpoint, some VCSs don't handle large binary files well, and others have strong opposition to including any sort of binary file in a VCS. If either of those cases apply to you, then storing the test files in a well known location that is easily accessed would also make sense.

I would also consider putting a comment in the test code that says "relies upon foo.pdf in order to run all tests."

  • I don't see anything wrong with having the tests check for the test data, if not found then trying to get it (eg. from a URL) and failing if neither worked. Relying on the network is a bad idea because it makes tests slower and more fragile; but trying is less fragile than not, and automatically getting (and caching locally) the right data is quicker than manually reading docs/comments, getting it and putting it in place. – Warbo Oct 2 '14 at 23:26
7

If it's static data, then yes put it in version control. Those files won't really change once they're checked in; they'll either get removed if that functionality's no longer needed, or new test files will be added alongside. Either way, you don't need to worry about poor binary diffs taking up space.

If you're generating test data, eg. randomly, then you should automatically save it when a test fails, but discard it otherwise. Any data saved this way should be turned into regular regression tests, so that those edge-cases are definitely tested in the future rather than relying on the luck of the draw.

  • 2
    If you're generating test data randomly, then you should really go out and buy a book about writing reproducible automated tests. – Dawood ibn Kareem Oct 2 '14 at 22:30
  • 5
    @DavidWallace So you're saying entire fields like fuzz testing, property checking and statistical software testing are not only wrong, but harmful? – Warbo Oct 2 '14 at 23:17
  • 5
    @DavidWallace random != unreproducible. – congusbongus Oct 3 '14 at 2:19
  • 5
    @DavidWallace you may call it whatever you want then. Random test data, record inputs, recycle if necessary, ergo reproducible. Doesn't lead to a world of hurt. – congusbongus Oct 3 '14 at 4:00
  • 2
    @DavidWallace "instead of stopping to think about which test cases are actually needed" doesn't mean "no random testing", it means "not only random testing". As for "you can't reproduce the data that found the bug", did you actually read the answer you're commenting on? ;) – Warbo Oct 3 '14 at 7:05
0

Definitely include that data with your tests and your main application code. It helps to have a really well organised test suite - so if you're testing pdf extraction (and you have that code nicely encapsulated) then you should be able to construct a path to your test data, based on the path to the app code - that's always worked for me.

With git you can set up a .gitignore to prevent any temporary output or test logging from polluting your repo.

protected by gnat Sep 13 '18 at 5:27

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.