9

I have smaller unit tests that use small snippets from real data sets. I would also like to test my program against full data sets for a multitude of reasons. The only problem is that a single real dataset is about ~5GB. I haven't found any hard numbers for what Git repositories can store but that seems like too much.

According to this Programmers post I should keep all of my data needed to test the project in the repository.

The solution that my team has adopted is that the project has a file that contains a path to a network attached file system that holds our test data. The file is git ignored.

I feel like this is an imperfect solution for two reasons. When the NAS isn't working, is slow, or is down than we can't run a full test. The second reason is that when someone first clones a repository the unit tests fail so they have to figure out how to mount things with a certain name and the syntax used to build the testing path file.

So my question is two fold. How much data is too much data to store in revision control?

What is a better way to handle large amounts of test data?

  • 1
    How often is the test data likely to change? – Robert Harvey Dec 8 '14 at 23:24
  • It will probably never change but more data might be added as we patch bugs or add features. – AlexLordThorsen Dec 8 '14 at 23:25
  • 1
    Some of the tradeoffs are explored here: stackoverflow.com/q/984707 – Robert Harvey Dec 8 '14 at 23:28
  • 1
    Irrespective of what git holds, have you considered it from the point of view that a full dataset from live data is not a test dataset (designed to test both success and failure states) and that alone may be a strong argument for it to be held outside the repository? – James Snell Dec 8 '14 at 23:56
  • Unit tests should not be using that much data. It's conceivable that integration tests might. – raptortech97 Dec 9 '14 at 14:51
9

How to handle large files in a build chain

I like to use a build tool that does dependency management - such as maven or gradle. The files are stored in a web repository, and the tool takes care of downloading and caching automagically when it encounters the dependency. It also eliminates extra setup (NAS configuration) for people who want to run the test. And it makes refreshing the data fairly painless (it's versioned).

What's too big to put in revision control

There is a large gray area. And if you decide something doesn't belong in a RCS, what are your alternatives? It's an easier decision if you limit your choices between the RCS and a binary repo (maven style).

Ideally, you'd only want in the RCS stuff that is humanely editable, diffable, or where you'd want to track the history. Anything which is the product of a build or some other sort of automation definitely doesn't belong there. Size is a constraint, but not the main one - a giant source file (bad practice) definitely belongs in the source control. A tiny compiled binary doesn't.

Be ready to compromise for developer convenience.

3

When the NAS isn't working, is slow, or is down than we can't run a full test.

Obviously, this can only be solved by copying the 5GB from the NAS to your local drive. But there is no need to do this manually.

The second reason is that when someone first clones a repository the unit tests fail so they have to figure out how to mount things with a certain name and the syntax used to build the testing path file.

You could provide a simple shell script which does exactly this - mount the NAS with a certain name, and copy the data to your local drive when it is not already there, or when the dataset at the NAS is newer than the local dataset. Make sure the script will run automatically during the initialization stage of your unit tests.

Of course, when there is not only one of those data sets, but a whole bunch of dependencies to external files outside of your source code repository, then a tool like the ones mentioned by @ptyx might be the better solution.

3

...when someone first clones a repository the unit tests fail so they have to figure out how to mount things with a certain name and the syntax used to build the testing path file.

First, just to have a consistent terminology: This kind of test (large external dependencies, real data) is usually not considered a unit test, but rather an integration or system test.

On a practical note: I find it a good practice to keep unit and integration tests separate, because they have different strenghts and weaknesses.

  • separate the two kinds of tests in the code (naming convention, separate project, ...)
  • provide a way to run only one of the two suites of tests
  • run only the unit tests during normal builds
  • run the integration tests on demand, and on a CI (continuous integration) server

That way, local builds are fast and reliable (little/no external dependencies), and integration tests are handled by the beefy CI server. This avoids the problem you describe.

As to how to keep the data:

One good option is some kind of artifact management like ptyx' answer describes. Another would be to put the test data into a separate repository. The data is not released together with the main build anyway, and having a separate repo avoids forcing everyone to fetch the test data along with the source code. In other words, use a second repo as your artifacdt management :-).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.