2

We have multiple projects that are using common binary assets (e.g. images, sounds, 3D models etc.) stored in a folder on a server. Right now, each project adds these assets to their git repository. The result is that we have multiple copies of the same asset across different repositories, and if the asset changes, each of the teams must update them manually from the folder on the server.

We would like to better organize this asset pool, so that each repository stores a reference to a specific asset, such that they can be updated more easily, and thus prevent asset duplication.

How to do this best using git? We thought about submodules, but this would mean creating a submodule for each asset, which doesn't scale well (we have thousands of assets, and each project uses 10-100 of them). Is there a way of creating a single repository with all assets, then selectively cloning some of them in each project repository, but maintaining the references to the asset repository so that they get updated when the original asset changes?

1 Answer 1

1

Instead of sticking your assets into git repositories, consider whether you can maintain a server of all assets, where each asset version has a stable URL (for example, the asset's hash, or asset name + publication date, or name + version number). Your versioned build scripts then contain the assets' URLs, and download the assets at build time (possibly cached). To update the assets, update the URLs in your scripts (You could write a tool that does this automatically!).

If you want to implicitly update the assets in all repos when a new asset version is uploaded, use an URL that always points to the most recent asset version. They will be downloaded by the build scripts for the next clean build. However, this would mean that you cannot build old versions with the assets they used at the time. This partially defeats the point of version control, so using stable URLs would be preferable.

Such asset management approaches are similar to dependency management: when my code has a dependency on another library I do not add that library as a submodule, nor do I copy library binaries into my repository, but I use a package manager to install the library. You could literally use an existing package manager that connects to a private server to manage your assets. A package manager can also help you with asset updates.

There are Git extensions that provide related functionality, e.g. Git LFS. LFS stores specified files outside of the Git history but will fetch them from an LFS server on demand. However, the intention here is not to manage shared assets but to keep the repository size small, so it might not be suitable for your use case.

Stepping out of Git means that my repository is no longer self-contained: I need a network connection to do a clean build. But you have to draw the line somewhere. E.g. most projects do not version-control their compilers or operating systems as part of the repository (even though virtualization technologies and containers make this feasible).

2
  • 1
    Thanks for the answer, the GIT LFS seemed to be perfect at first glance, but on further reading looks only suitable for the purpose you've just mentioned. Your idea is interesting, unfortunately not suitable for our case. Our IDE (game engine) needs the assets at all times, not only at compile time. This would mean having those scripts running when committing/updating things, which would make the workflow too complicated.
    – John S
    Oct 16, 2018 at 15:16
  • @JohnS Possibly git hooks which can run after each checkout could make a workflow more tolerable. But yeah, I can see how your problems might be more complicated than that. My idea with build time scripts was that the build produces an artifact that is subsequently run/tested, where “build” incorporates more than just compilation but also things like updating an asset.
    – amon
    Oct 16, 2018 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.