Binaries in source control

Question

When developing for embedded devices and other odd worlds, it's very likely your build process will include multiple proprietary binaries, using very specific versions of them. So the question is, are they part of your source control? My offices goes by the rule of "checking out from source control includes everything you need to compile the code" and this has led to some serious arguments.

The main arguments I see against this is bloating the source control DB, the lack of diffing binary files (see prior questions on the subject). This is against the ability to check out, build, knowing you have the precise environmental the previous developer intended and without hunting down the appropriate files (with specific versions no less!)

Answer 1

The idea of VERSION CONTROL (misnomer: source control) is to allow you to roll back through history, recover the effect of changes, see changes and why made. This is a range of requirements, some of which need binary thingies, some of which don't.

Example: For embedded firmware work, you will normally have a complete toolchain: either a proprietary compiler that cost a lot of money, or some version of gcc. In order to get the shipping executable you need the toolchain as well as the source.

Checking toolchains into version control is a pain, the diff utilities are horrible (if at all), but there is no alternative. If you want the toolchain preserved for the guy who comes to look at your code in 5 years time to figure out what it does, then you have no choice: you MUST have the toolchain under version control as well.

I have found over the the years that the simplest method to do this is to make a ZIP or ISO image of the installation CD and check this in. The checkin comment needs to be the specific makers version number of the toolchain. If gcc or similar, then bundle up everything you are using into a big ZIP and do the same.

The most extreme case I've done is Windows XP Embedded where the "toolchain" is a running Windows XP VM, which included (back then) SQL Server and a stack of configuration files along with hundreds and hundreds of patch files. Installing the whole lot and getting it up to date used to take about 2-3 days. Preserving that for posterity meant checking the ENTIRE VM into version control. Seeing as the virtual disk was made up of about 6 x 2GB images, it actually went in quite well. Sounds over the top, but it made life very easy for the person who came after me and had to use it - 5 years later.

Summary: Version control is a tool. Use it to be effective, don't get hung up about things like the meaning of words, and don't call it "source control" because its bigger than that.

Answer 2

Neal Ford argues in The Productive Programmer that you should keep binaries in source control:

Why keep binaries? Projects today depend on a swath of external tools and libraries. Let’s say you are using one of the popular logging frameworks (like Log4J or Log4Net). If you don’t build the binaries for that logging library as part of your build process, you should keep it in version control. That allows you to continue to build your software even if the framework or library in question disappears (or, more likely, introduces a breaking change in a new version). Always keep the entire universe required to build your software in version control (minus the operating system, and even that is possible with virtualization; see “Use Virtualization,” later in this chapter). You can optimize retaining binaries by both keeping them in version control and on a shared network drive. That way, you don’t have to deal with them on an hourly basis, but they are saved in case you need to rebuild something a year later. You never know if you will need to rebuild something. You build it until it works, then forget about it. It is panic inducing to realize you need to rebuild something from two years ago and don’t have all the parts.

I couldn't agree more; while this is arguably subverting the VCS for a task it wasn't designed for ( keeping binaries ), I think the benefits outweigh the potential drawbacks. But, as the author notes later, sometimes keeping the binaries in VCS might not be a practical solution, so other options should be considered - like keeping them on a mapped network drive.

If the binaries aren't too big, I would definitely keep them in VCS. This seems to be even more true in your case, since the binaries are probably small, and you work with very specific versions. They might also be hard to find, due to a variety of reasons ( the authors shut down their website, or the version you need is no longer listed for downloading ). Although unlikely, you never know what will happen in a few years.

I wish I read this book a few years ago, when I was working on a game using a graphics library ( which was dll file ); I interrupted the development for a while, and when I wanted to continue I couldn't find the dll again because the project died.

Answer 3

Source control is for sources. Sources are what you're unable to build from other things. Some files that qualify as sources happen to be binaries.

My VCS has lots of binaries checked into it, but each one is the unit of release from some product I didn't write and don't maintain. This might be something like GNU ccRTP, which is released as a compressed tarball. That tarball is my source, and it's checked in along with whatever infrastructure I need to turn it into a finished product (a Makefile and an RPM spec in my case) in a single, automated step. When there's a new version of ccRTP, I treat the new tarball as changed source: it goes into a checked-out copy, gets built, tested and committed back to the VCS. I've done the same with commercial products that don't ship with source (compilers, libraries, etc.) and it works the same way. Instead of unpack-configure-compile-package, it's just unpack-package. The software that does the nightly builds doesn't know or care as long as it can run make and get finished products.

Most VCSes have features that make human-readable source easier to deal with and more efficient to store, but to say that they aren't suited to binaries isn't really true if binaries put in come back out unmolested. How a VCS deals with binaries internally depends entirely on whether or not its authors thought attempting to only store differences was worth the effort. Personally, I think storing full copies of a the ccRTP distribution at 600K a pop is more than made up for by the ability to tag a version of it along with all of my other sources.

Answer 4

In principle, I appreciate the "check everything you need to build into source control" camp, but dependency management has evolved quite a bit in the last few years, with tools like Maven, Ivy and NuGet.

Also, in practice, I find checking in binaries to create a number of unpleasant side effects. Git/Mercurial aren't really tuned for it, for example, and Subversion and Perforce can drive you nuts when merging branches that contain binaries.

With a dependency management solution, you specify in a source-controlled file in your project which package names and which versions your project depends on. Almost all dependency management tools allow you to create a private repository of your dependencies, following some sort of versioning and naming convention; when you do a build, the dependency management tool will resolve all of your open source and proprietary dependencies from a list of approved sources, then stuff them into your local cache. Next time you build with the same version dependencies, everything's already there and it goes much faster.

Your private repository can then be backed up with conventional filesystem backup tools.

This avoids the slowdowns I've experienced when a ton of binaries are being pulled from the source tree, and prevents your repository from having lots of hard-to-diff files. There's only one location for any given dependency, by name and version number, so there's no merge conflicts to deal with, and the local filesystem caching means that you don't have to deal with the cost of evaluating whether your local copy has changed when you pull updates.

Answer 5

This reminds me of the "jars in repository" problem that some time ago Java had. People building java apps were used to push their dependencies (binary jar files) into repositories. Everybody were happy with this, because we you would have "one click" build system and disk space is cheap, so who cares. Then came Maven and you could get rid of all that binary cruft and with local cache-only repository still maintain bullet-prof builds. Still you have "one click" build system, but source control doesn't have to shuffle around binary files that make no sense there.

So yeah, you can get binary files out of the source control, but this will require you to tweak the build system, to get them at build time. Without dedicated software (like Maven) this might be a lot of effort to just get them out.

Answer 6

Your source control hold the sources to what you do. If a given binary blob can be reconstructed from the sources it is not a source and should not go in the source code repository. Only non-recreatable blobs should to in the source control.

You usually have another repository network folder of binary blobs you've built through time of the sources. These can be deployed to customers or used in projects (instead of building everything from scratch every time).

So, put it in if it is a source. Don't if not.

Answer 7

The goal is to be able to get the latest code and build it without having to install/setup anything (so, a "single click" build).

In many places I have been, that means checking in binaries of dependencies. In others, this means that the build scripts download and get the dependencies automatically.

See this blog post by Derek Greer on the subject.

Answer 8

I am working at a project with two different build stages

• the "main program build" needs just a few binaries, compared to the thousands of source code text files, so the binaries are checked into the repository. This works fine.

• the installer build needs a lot of third party components (some of them are just copied to the installation CD, like the Adobe Reader). We are not putting those into the repository. Instead, those components reside on a network drive (even older versions of them), and the build scripts copy them to the right place. Of course, to have reproducible builds, anyone has to be careful not to change any folder where the third party components are stored.

Both strategies work fine and fulfil the "checking out from source control includes everything you need to compile the code" requirement.

Answer 9

You need to keep everything you will need to rebuild specific versions of the product at some point in the future.

However you don't have to keep everything in Source Control.

One company kept a frozen server rack (because the OS only ran on that specific hardware, and the toolchain only ran on that OS, and the source was dependent on that toolchain). Can't check that into Source Control.

If you do need to split up the requirements for a build, then you have the accounting problem of keeping two version control systems sync'd. e.g. the hardware box in this closet, or the VM or binaries in this preserved backup volume, go with this SVN Source Code revision, etc. This is messier that using a single source control system, but solvable.

Answer 10

It's very chaos to check-in binary to SCM in my mind. I had run a very complex project, which has a lot of dependencies to third part libraries. The principles that we adopted:

1. All source code is managed with SCM
2. All dependencies are managed with Ivy, which has great eclipse integration.

This works pretty well. We have a configuration file about version of each external library that source code can be compiled with. This configuration file is checked into SCM, so it evolves as source code evolves. By applying this approach, we can exactly reproduce a build without messing around version of external libraries.

Answer 11

Personally, philosophically, I am inclined to let source control check in pointers to the large binary files (small binary resources are OK), and not the contents of the file. This pointer would contain a hash of the binary file contents.

The binary file itself would not be managed by source control. It would be stored in some kind of library where it can be retrieved using the pointer, or the hash specifically.

Git LFS and git annex do that, but they also try to manage the binary files to some extent, I don't want them to do that. I want Git to store checksums only, and to tell me if my binary files have changed or not - but I do not want it to try to manage them and to store them. I want to do this myself.

I think that git can handle small and medium sized binary files but I am not sure that it is the right tool for managing large binary files.

Answer 12

Repeatable Builds

The underlying premise of the question is that the product getting built requires repeatable builds.

Why should you care about repeatable builds? In general, that question deserves a thread of its own. But one common use case is the hot fix.

Say you have a customer running 5 year old version of your product Foo-Widget v3.2.1. Your most recent release was v5.4.3 and you stopped supporting 3.x last year. But, ok, the customer hits a bug, does not want to upgrade and is waving a checkbook at the sales team to get them a fix. You get marching orders to deliver that fix. How does that work?

From the source code perspective, you have a tag Foo-Widget-v3.2.1 you can use to check out the exact sources. So, that's cool. But some bright person said, "thou shalt not check in binaries with source code", and your build system was updated to new compilers, etc. with version 4.x. Hmmm ...

One good approach would be to use container images to build your products. These images have source code of their own kept in a separate module/repo and have a build of their own. The result is a container image stored in your local container image repository. Your build system executes builds by firing up a fresh container from the image in question and running your build script. So, in the product source tree, the build script keeps a reference to the container image required to build it. Note, developers should be able to run the same image on their development machines.

So, all inputs to the build come from controlled sources and the artifacts generated, likewise, store the products in a shared repository. Welcome to the wonderful world of DevOps.

There are other ways to achieve that result without resorting to checked in binaries - as described by several other good answers to this question. They all involve controlled, long term storage for your build and runtime artifacts and referring to those binaries within the product source code.

Even if you just have an old-fashioned "build machine", with all the required tools installed, where your official builds are performed, there is another simple approach: backups. After updating the build system, simply take a full backup and keep it in a very safe place. When the requirement to build a hotfix for an ancient release comes in, stand up a new build system and restore the backup to it. Run your build.

Point is there are many solid ways to implement repeatable builds without trashing your source code.

Problems with Checked In Binaries

Some problems with checked in binaries include:

• Lack of provenance
• Sclerosis of the source code
• Change analysis is difficult

Lack of Provenance

Where did the binary come from? How do we know it is clean from malware, spyware, worms, etc.? Artifact repositories, like Artifactory, Maven repos, container repos have audit logs, keep hash codes of every binary. For replicated artifacts, there is a clear chain of custody from the original source. These are all basic requirements for organizations that care about their own security and that of their customers.

Sclerosis of the Source Code

It takes a looong time to clone the repo. Even switching branches can slow down. The slowdown can especially affect teams in remote offices. Developers' time matters, too.

Change Analysis is Difficult

Binary files are not comparable in any useful way. Instead, you need to refer to the release notes, change log and even source code of the product. Having links back to the full source, where possible, is really important for understanding what goes into a new binary. Checked in binaries can easily lose all of that important information.