6

I'm unit testing some code that processes data from an external API. That external API serves up a large pile of JSON data.

The right thing to do here, of course, is to use fixtures in the unit tests, so that I don't do a real API call every time the test runs.

Those fixtures can be quite large though - many thousands of lines, even. I'm not sure whether I should put them into git. It seems like a bad idea, since they seem like static files. On the other hand, writing the unit tests really requires specific fixtures, so I do want them shared with anybody working on the project (and shared with our build machinery).

Should large fixtures be committed into a repo?

Improve this question

3 Answers 3

Reset to default
12

Absolutely yes. Anyone should be able to run your unit tests by getting them out of the repository. Any build automation is going to require this (it should, anyway), as will future developers.

An alternative might be to put a pre-test block in the unit test that checks for the files and downloads them if they're not there. That way it's easy for you to update them if the API changes, and anyone getting the code will just be able to run it without digging round your machine to find the missing files.

As Guy pointed out, compressing files is unlikely to help - between the SSD/disk cache and git compressing things you almost certainly won't see a speed increase. With large chunks of static data it might make sense to put a gz wrapper around them (or similar) to speed up loading if that's easy to do in your language. That pushes the size down, and with fairly static data new commits (and consequent diffs) are going to be very rare.

Improve this answer
1
  • The answer and reasons given are correct. I would suggest considering a submodule for very large projects, though – downloading the fixtures is still as easy as git pull && git submodule update, and you don't have to worry about inflating your primary codebase. That said, submodules have a bit of a learning curve, and wouldn't be worth it for small projects.
    – Don McCurdy
    Commented May 8, 2014 at 21:14
4

Unless you're talking about terabytes of data (which I don't believe you are here) then I would put it in source control. You really want everything in your source control solution so that anybody can check it out and run the full test suite without requiring any other dependencies to be hooked up.

I would also not zip the data as many source control systems already provide compression behind the scenes and if not the source control system then the media that you store it on may provide compression. The reason that I would generally not compress data is that there are times when you'll want to change the data and you'll want to be able to do a diff between different versions. Say the data returned by the API changes. That's a good reason to change the fixture's data.

Improve this answer
1

I would commit them to some sort of a repository, yes.

It doesn't need to be the main repository, but you want to preserve the effort you put into creating the fixtures in the first place. Likewise, if someone else works on the project with you, then being able to check out those fixtures will save them a lot of time.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.