I'm a big fan of using open source software but I mostly use community binary releases for the job.

I'm wondering about companies that go the extra degree and build everything they use in their production server application infrastructure entirely from source, in-house, in an effort to promote transparency and supportability (instead of relying on binaries). Is it practical and sustainable to do this?

It seems that the sheer number of dependencies to maintain and build from source for some of the major open source projects would quickly complicate things.

  • 1
    I understand that Google does this. Are they unique? From what I've researched, it takes a lot of maintenance but it is sustainable because their applications are structured with a level of homogeneity: they are written in either C, Java, Go, JavaScript, and Python and with strict coding guidelines. The infrastructure required to support their applications relies on only a few third party components (even Java). – Brett Bonner Jan 10 '13 at 22:12
  • 1
    This is a giant "it depends on what you mean". Are you talking about the dev machine software? Or the products themselves? Or the infrastructure servers? – Paul Nathan Jan 10 '13 at 22:30
  • "Software infrastructure stack" = Production application server infrastructure. For example, building an application server, building monitoring tools, building the OS (kernel, userland stuff, etc) – Brett Bonner Jan 10 '13 at 22:33
  • 2
    If google's servers were all running on their own kernel I would be thoroughly flabbergasted. That sounds ridiculous, no one does this to my knowledge other than companies which write their own OS to begin with (mainly hardware manufacturers, apple, and MS). – Jimmy Hoffa Jan 10 '13 at 23:03
  • @JimmyHoffa oh, I see how you're confused. I certainly don't mean they code everything from bare-metal up from scratch. Not even Microsoft does that. I mean, are there any companies that run a software infrastructure platform that is built (compiled, linked, tested, packaged) in-house from source versus relying on community binary distributions. I do know that Google's servers run their own Linux distribution with kernel patches, which of course they do build from source. All the code is in Perforce along with Gmail code and everything else they write and it's all tested continuously. – Brett Bonner Jan 10 '13 at 23:16

Among the hats I wear is overseeing packaging and installation of the software and the development tool set for one of my company's business units. We build everything (almost; see below) from source from scratch nightly using a custom-built, fully-automated process. Anything not burped out by the build process doesn't ship, no exceptions.

We do this for a lot of reasons:

  • Safety. Building from source mitigates the risk that a provided binary has had something malicious added. If we suspect software-induced foul play, we can go back and audit the sources.

  • Accountability. If we change versions of a package, we'll know about it because we put the distributions (usually as the blob, not broken out into individual files) into our VCS.

  • Maintainability. As we correct bugs or make enhancements, we write patches that go into our VCS with the distributions and get applied as part of the build process. As we upgrade to later versions, we remove patches if what they do has been incorporated into the source and review the remainder for relevance. Patching also adds accountability we wouldn't have otherwise ("that broke after we applied the autodefrobulator patch").

  • Insight. Because we go through the process of building sources into packages, we get a good understanding of the dependencies and what quirks there are in the build process. Because builds are automated, the known-correct process for building the package is scripted and can be read and understood. In at least one case, we've ditched something because the process of figuring out how to make it build caused us to look closely at the source, when we discovered that the entire thing was poorly written.

There are two things we don't build:

  • Operating Systems. The benefit we would get from building the entire thing from source isn't worth the administrative effort. We get our OS media from sources we trust, usually directly from the vendor, and treat that as a unit. Most of our OS installations are done from a central server, so we keep that up to date with the latest patches.

  • Commercial Products. There are a handful of things we use that are either unavailable or prohibitively expensive in source form. In those cases, we keep the binary distributions, one for each architecture, in our VCS with everything else. In some cases, when the vendor provides it in something other than our preferred installation format, we'll actually have a build process that repackages it as we want it. In all cases, even if the build process is "do nothing, here's your finished package," these go through our automated build process.

    Is it practical and sustainable?

Absolutely. There is a certain level of labor and equipment required to get it started and the kinks worked out, but our process has been set-and-forget. I haven't done a detailed accounting of it, but I'm pretty sure we spend a lot less time downloading and compiling software because the engineering staff can go look in a well-known place for what they need. Those of us that do the actual packaging have become are good enough at it that most new additions are available after the next nightly build.

We're not doing anything on the scale of spinning a Red Hat distribution, which involves 3,000+ packages. We're an order of magnitude smaller than that, but my experience has been that once the figure gets beyond a certain point, consolidating everything into a central system is worthwhile.

It seems that the sheer number of dependencies to maintain and build from source for some of the major open source projects would quickly complicate things.

If your builds are done using a packaging system that can handle dependencies and conflicts (e.g., RPM), it's not as difficult as you might think. We do go through "dependency hell" when we build something that has a lot of prerequisites that are new to us, but once it's done, it's done. After that, the nightly build takes care of making sure everything is in the right order. Understanding the dependencies forces us to think about the overall architecture and how new additions fit in. We'll reconsider something if it's going to turn into a rat's nest.

Addendum based on comments:

Multiple Versions: I go to great lengths to avoid situations where we need multiple versions of the same thing installed, but sometimes the ugly train stops at your station and you have to board it if you want to get anywhere. Python has been a bit of a thorn in my side in that respect, because we get a few things from other parts of the company where they've used features from later versions than the one we like to use. Fortunately, Python is built so multiple versions can be installed in parallel, so we do that and write patches for the "outside" software to force it to use the right one. On other occasions, I've had to do full repackagings so software requiring a special version of other software has the what it needs bundled in and isolated from the rest of the system.

Upgrades: I don't do them unless they need to happen. Once nice thing about building from sources is that you can pick and choose things from later versions if you need them and turn them into patches to your existing version. I do this with bug fixes.

| improve this answer | |
  • Well written and extremely helpful! So regarding maintenance of dependencies, especially when it comes to third party software libraries, is your policy to support multiple revisions (versions) of those libraries or does your team make an effort to use your latest build available of those dependencies? For example, if Tomcat needs one version of commons-lang and ActiveMQ needs another. Do you schedule refreshes to update your artifacts from time to time by merging your code with the community or is it more of a "update it when the build or test breaks" thing? – Brett Bonner Jan 11 '13 at 1:00
  • @LaughNowButWe'llBeInCharge: See the addendum to my answer. – Blrfl Jan 11 '13 at 2:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.