This question already has an answer here:

We have a camera application that does measurements on pictures. We have a feature for saving a "demo" containing the pictures/data used, and to replay this for test purposes. We use such data sets for different tests. We currently have these data sets stored on a server, accessing them directly in the file system.

I am currently working on a test suite where we run large amounts of demos (~500GB data) to ensure stability in measurement results. Each test case is a set of demos, along with setup information for the test framework and software.

Getting the data is currently done by copying the contents on the server over to the test computer once in a while. This takes a lot of time, and is less than perfect (doesn't delete files on computer if they are deleted on server..)

I also very much want the setup information in version control. We have already had issues where changing the setup-data has been a headache.

Is it possible to use git for this? It appears to solve both problems, simply call git pull. It doesn't seem right to add large amounts of binary data to git, tho.

marked as duplicate by gnat, Becuzz, Robbie Dee, Thomas Junk, 9000 Apr 5 '18 at 15:09

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • If you are worried about the size, you can always maintain a separate repository for the test data and keep it linked as a git submodule. – isaach1000 Apr 5 '18 at 14:05
  • 1
    Why not using linux.die.net/man/1/rsync ? – Thomas Junk Apr 5 '18 at 14:36
  • If you want a kind of versioning that git provides, but with an external storage, consider git-annex. – 9000 Apr 5 '18 at 15:09
  • Thank you for the link to the duplicate question. Re-reading it and giving it some time has really helped. I think my question now boils down to "How much data can git handle?" – Petter TB Apr 6 '18 at 9:27

It depends on where your database environment is located. If it was located in AWS then a possible answer is to have an S3 bucket under which the immediate sub-folders are named for the git version number that their data represent. What goes under source control is the name of the subfolder within the bucket and the bulk load statements to load your database. There are some disciplines necessary to make this work. * Old versions have to be regarded as immutable. * A failed data load = a failed build. As you are doing, make sure there are tests to ensure the loaded data is as expected even if no explicit errors are thrown.

Not the answer you're looking for? Browse other questions tagged or ask your own question.