3

The codebase consists of 190 Fortran 95+ source files, and directory of binary coefficient files required for the use of the code. The binary directory contains about 4000 files, some of which are greater than 100 mb, but no larger than 200 mb. The vast majority are less than a megabyte. The files do get updated occasionally, and each semi-major release of the code 1.x requires a recalculation of all of the coefficient files.
Multiple users will be accessing the code through the process of the development cycle, with multiple branches.

The question is how do I appropriately manage and distribute this code base using Git? Before marking this duplicate, I've read carefully through the various StackOverflow answers to this question. Git-lfs appeals to me the most, but the development environment that I have does not want to enable git-lfs support (federal government operational computers).

Actually native git handles the files reasonably well, and my diffs and merges don't take an absurdly long time -- but it's a rather abusive way to go about it in terms of disk space required for the git internal storage.

I do also want to be able to host on bitbucket or, preferably, github. Github has a 100 megabyte file size limit, IIRC. So whatever the solution, it needs to be compatible with online hosting solutions.
I realize this may be restrictive, but this is also why I'm asking.

Kind regards,

-Ben

Improve this question
10
  • 2
    have you thought about git large file storage, git-lfs.github.com
    – DFord
    Dec 28, 2017 at 16:21
  • Thanks for your comment. git-lfs is not available to me on my development server, but there's no reason I cannot use it for the community release. I think I may have to consider that possibility.
    – Ben
    Dec 28, 2017 at 16:35
  • These coefficient files – if they can be recalculated, why do they need to be under version control? Shouldn't you only commit the programs to generate these files, then run them as part of the build process? Or does it take too long to recalculate these coefficients?
    – amon
    Dec 28, 2017 at 17:04
  • And why do the coefficient files have to be updated for each release? Do they change because your code has switched to a different data format, or do they represent some external changing data? If it's just the format which changed, why can't you create a stable format?
    – amon
    Dec 28, 2017 at 17:04
  • 1
    Amon: Good questions: The coefficient files take from minutes to days to recreate, and cannot be accomplished by users (requires specialized / proprietary software and export controlled information, etc). Whenever there's a semi-major update (every 3-4 years), this usually impacts the various coefficient files. The format usually doesn't change, the contents do.
    – Ben
    Dec 28, 2017 at 17:07

1 Answer 1

Reset to default
2

Best approach I can think of is:

  • do not put the large binary files directly into version control, instead, put them in a folder with a version number on a network share (or whatever kind of file storage server is available)

  • add a script to your source tree which makes a local copy of the "current version" of the binary files from the network share into the local development environment, if it is not already there. Put that script under version control.

If required, you can also make the execution of that script part of your regular build process. At some complexity, utilizing a package manager for it might be a good idea, as mentioned by @BerinLoritsch. However, this will only make sense if you can pick the tools of your development environment freely.

That way, you can decouple the configuration management of your source code to some degree from the configuration management of the large binary files.

Improve this answer
3
  • Its stuff like this that we have package management systems these days. Binaries tend to make your version control balloon in size very quickly when you update them.
    – Berin Loritsch
    Dec 28, 2017 at 22:26
  • 2
    @BerinLoritsch: thanks, added a sentence to my answer. Package managers are providing many more features, they are most useful if one has to add lots external libs and tools into the build process. But they also come for a cost, they have a learning curve and their configurations need some maintenance effort.
    – Doc Brown
    Dec 29, 2017 at 8:18
  • I like this comment, but package management is platform dependent, whereas version control generally isn't platform dependent.
    – Ben
    Dec 29, 2017 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.