Reusing binary assets using Git

Question

We have multiple projects that are using common binary assets (e.g. images, sounds, 3D models etc.) stored in a folder on a server. Right now, each project adds these assets to their git repository. The result is that we have multiple copies of the same asset across different repositories, and if the asset changes, each of the teams must update them manually from the folder on the server.

We would like to better organize this asset pool, so that each repository stores a reference to a specific asset, such that they can be updated more easily, and thus prevent asset duplication.

How to do this best using git? We thought about submodules, but this would mean creating a submodule for each asset, which doesn't scale well (we have thousands of assets, and each project uses 10-100 of them). Is there a way of creating a single repository with all assets, then selectively cloning some of them in each project repository, but maintaining the references to the asset repository so that they get updated when the original asset changes?

Answer 1

Instead of sticking your assets into git repositories, consider whether you can maintain a server of all assets, where each asset version has a stable URL (for example, the asset's hash, or asset name + publication date, or name + version number). Your versioned build scripts then contain the assets' URLs, and download the assets at build time (possibly cached). To update the assets, update the URLs in your scripts (You could write a tool that does this automatically!).

If you want to implicitly update the assets in all repos when a new asset version is uploaded, use an URL that always points to the most recent asset version. They will be downloaded by the build scripts for the next clean build. However, this would mean that you cannot build old versions with the assets they used at the time. This partially defeats the point of version control, so using stable URLs would be preferable.

Such asset management approaches are similar to dependency management: when my code has a dependency on another library I do not add that library as a submodule, nor do I copy library binaries into my repository, but I use a package manager to install the library. You could literally use an existing package manager that connects to a private server to manage your assets. A package manager can also help you with asset updates.

There are Git extensions that provide related functionality, e.g. Git LFS. LFS stores specified files outside of the Git history but will fetch them from an LFS server on demand. However, the intention here is not to manage shared assets but to keep the repository size small, so it might not be suitable for your use case.

Stepping out of Git means that my repository is no longer self-contained: I need a network connection to do a clean build. But you have to draw the line somewhere. E.g. most projects do not version-control their compilers or operating systems as part of the repository (even though virtualization technologies and containers make this feasible).