52

I'm writing some test code for a feature which processes PDF files. The basic idea behind the tests is that I point them towards some PDFs I've selected specially, they process them and I check that the output is what I expect.

My question is: where should I be storing these large-ish PDFs? Should I check them into version control along with the code? Or put them somewhere else? Obviously, the test code is useless without the PDFs (or even with different PDFs) but still, putting them into our repository feels wrong.

Improve this question
3
  • 2
    possible duplicate of Should unit tests be stored in the repository?
    – user
    Commented Oct 2, 2014 at 20:30
  • 20
    @MichaelKjörling: Tests != Test Data
    – Robert Harvey
    Commented Oct 2, 2014 at 20:42
  • 6
    @RobertHarvey True, but if the test data is required for the test to work, I feel it should be considered a part of the test. That is also the approach taken by all three answers so far, as I understand them.
    – user
    Commented Oct 4, 2014 at 13:38

4 Answers 4

Reset to default
88

Your version control system should contain everything it needs to build, compile, test, and package an application for distribution (e.g. MSI, RPM). I would also argue build configurations and other scripts should also be in version control.

I should be able to check out a project and have a complete compile, build, and test environment.

There are two approaches to checking in test data. First, you can check in the test data itself (PDFs in this case). Second, you can check in source data that can be used to generate test data (if applicable). This could be a SQL script loaded into a blank database containing test data, or maybe a text-based file that can be compiled into a PDF or other file.

Others may disagree with checking everything into version control, but I have found in my professional experience it is critical to ensuring a complete environment is able to be rebuilt from scratch.

Improve this answer
12
  • 21
    Yes. Absolutely yes. It's 2014, there is no justification whatsoever for using revision control that doesn't handle binary files seamlessly.
    – Kilian Foth
    Commented Oct 2, 2014 at 17:38
  • 4
    I agree, but you definitely want to avoid the situation where you are checking in junk items as well. For example, if the test data includes an "output" folder that contains all the pdf files generated by the tests, then you will want to not include that into repository. But I do agree the tests themselves should be part of the repo as well as any packages needed to run it.
    – Kenneth Garza
    Commented Oct 2, 2014 at 19:37
  • 1
    @KennethGarza It isn't hard, really. As a rule of thumb, any original content (source code, test source code, test data, media, [real] documentation, third party libraries, build scripts, tooling scripts, conversion scripts, etc.) should be included, while all data that can be generated in reasonable time from the original data should not be. Besides, given those are the test outputs, they probably only make sense after running the tests yourself, otherwise you are not testing your program, you are testing the VCS software's ability to preserve the integrity of your files :)
    – Thomas
    Commented Oct 2, 2014 at 20:09
  • 2
    @MarnenLaibow-Koser: a project I worked on to detect electrical failure in implanted pacemaker leads had a test suite of over 40GB. There isn't a VCS in existence where dealing with that isn't obnoxious. Having two repos is an administration hassle of its own, but it sometimes can be the better choice.
    – whatsisname
    Commented May 18, 2018 at 2:40
  • 1
    @MarnenLaibow-Koser you got it. Integration tests are in separate repo and if user wants to run it locally, dependency management will fetch zip file for him and decompress it. Usually Continuous Integration server/farm is tasked to do integration test and will prevent merge feature branch until integration tests pass.
    – user482745
    Commented Nov 22, 2018 at 13:59
15

If the tests are useless without the setup files that you have prepared, then it makes sense to include the files in your VCS along with the test code.

While the files used in the test aren't code, you can view them as a dependency that the code relies upon. So there is merit in keeping everything together.

As a counterpoint, some VCSs don't handle large binary files well, and others have strong opposition to including any sort of binary file in a VCS. If either of those cases apply to you, then storing the test files in a well known location that is easily accessed would also make sense.

I would also consider putting a comment in the test code that says "relies upon foo.pdf in order to run all tests."

Improve this answer
1
  • I don't see anything wrong with having the tests check for the test data, if not found then trying to get it (eg. from a URL) and failing if neither worked. Relying on the network is a bad idea because it makes tests slower and more fragile; but trying is less fragile than not, and automatically getting (and caching locally) the right data is quicker than manually reading docs/comments, getting it and putting it in place.
    – Warbo
    Commented Oct 2, 2014 at 23:26
7

If it's static data, then yes put it in version control. Those files won't really change once they're checked in; they'll either get removed if that functionality's no longer needed, or new test files will be added alongside. Either way, you don't need to worry about poor binary diffs taking up space.

If you're generating test data, eg. randomly, then you should automatically save it when a test fails, but discard it otherwise. Any data saved this way should be turned into regular regression tests, so that those edge-cases are definitely tested in the future rather than relying on the luck of the draw.

Improve this answer
17
  • 5
    @DavidWallace So you're saying entire fields like fuzz testing, property checking and statistical software testing are not only wrong, but harmful?
    – Warbo
    Commented Oct 2, 2014 at 23:17
  • 5
    @DavidWallace random != unreproducible.
    – congusbongus
    Commented Oct 3, 2014 at 2:19
  • 6
    @DavidWallace you may call it whatever you want then. Random test data, record inputs, recycle if necessary, ergo reproducible. Doesn't lead to a world of hurt.
    – congusbongus
    Commented Oct 3, 2014 at 4:00
  • 4
    @DavidWallace "unreproducible is EXACTLY what random means" - ludicrous. It's trivial to record and replay that data later, as Warbo points out. And also as he points out, random input testing makes an excellent complement to regular testing, not a replacement for it. Running specific, thought-out test cases is a great practice which isn't going anywhere, but being human, we will err and miss some.
    – Chris Hayes
    Commented Oct 3, 2014 at 7:11
  • 3
    @David If you really think that fuzz testing is useless you obviously don't work in any field that is security sensitive. Fuzz testing is one of the best ways to find serious bugs in critical software. It's used to test compilers, browsers, OSes and many other things.
    – Voo
    Commented Oct 3, 2014 at 21:20
0

Definitely include that data with your tests and your main application code. It helps to have a really well organised test suite - so if you're testing pdf extraction (and you have that code nicely encapsulated) then you should be able to construct a path to your test data, based on the path to the app code - that's always worked for me.

With git you can set up a .gitignore to prevent any temporary output or test logging from polluting your repo.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.