Where should I store test data?

Question

I have smaller unit tests that use small snippets from real data sets. I would also like to test my program against full data sets for a multitude of reasons. The only problem is that a single real dataset is about ~5GB. I haven't found any hard numbers for what Git repositories can store but that seems like too much.

(According to this Programmers post I should keep all of my data needed to test the project in the repository.)

The solution that my team has adopted is that the project has a file that contains a path to a network attached file system that holds our test data. The file is Git ignored.

I feel like this is an imperfect solution for two reasons. When the NAS isn't working, is slow, or is down than we can't run a full test. The second reason is that when someone first clones a repository the unit tests fail so they have to figure out how to mount things with a certain name and the syntax used to build the testing path file.

So my question is two fold. How much data is too much data to store in revision control?

What is a better way to handle large amounts of test data?

Answer 1

How to handle large files in a build chain

I like to use a build tool that does dependency management - such as maven or gradle. The files are stored in a web repository, and the tool takes care of downloading and caching automagically when it encounters the dependency. It also eliminates extra setup (NAS configuration) for people who want to run the test. And it makes refreshing the data fairly painless (it's versioned).

What's too big to put in revision control

There is a large gray area. And if you decide something doesn't belong in a RCS, what are your alternatives? It's an easier decision if you limit your choices between the RCS and a binary repo (maven style).

Ideally, you'd only want in the RCS stuff that is humanely editable, diffable, or where you'd want to track the history. Anything which is the product of a build or some other sort of automation definitely doesn't belong there. Size is a constraint, but not the main one - a giant source file (bad practice) definitely belongs in the source control. A tiny compiled binary doesn't.

Be ready to compromise for developer convenience.

Answer 2

When the NAS isn't working, is slow, or is down than we can't run a full test.

Obviously, this can only be solved by copying the 5GB from the NAS to your local drive. But there is no need to do this manually.

The second reason is that when someone first clones a repository the unit tests fail so they have to figure out how to mount things with a certain name and the syntax used to build the testing path file.

You could provide a simple shell script which does exactly this - mount the NAS with a certain name, and copy the data to your local drive when it is not already there, or when the dataset at the NAS is newer than the local dataset. Make sure the script will run automatically during the initialization stage of your unit tests.

Of course, when there is not only one of those data sets, but a whole bunch of dependencies to external files outside of your source code repository, then a tool like the ones mentioned by @ptyx might be the better solution.

Answer 3

...when someone first clones a repository the unit tests fail so they have to figure out how to mount things with a certain name and the syntax used to build the testing path file.

First, just to have a consistent terminology: This kind of test (large external dependencies, real data) is usually not considered a unit test, but rather an integration or system test.

On a practical note: I find it a good practice to keep unit and integration tests separate, because they have different strenghts and weaknesses.

• separate the two kinds of tests in the code (naming convention, separate project, ...)
• provide a way to run only one of the two suites of tests
• run only the unit tests during normal builds
• run the integration tests on demand, and on a CI (continuous integration) server

That way, local builds are fast and reliable (little/no external dependencies), and integration tests are handled by the beefy CI server. This avoids the problem you describe.

As to how to keep the data:

One good option is some kind of artifact management like ptyx' answer describes. Another would be to put the test data into a separate repository. The data is not released together with the main build anyway, and having a separate repo avoids forcing everyone to fetch the test data along with the source code. In other words, use a second repo as your artifacdt management :-).