74

Due to a number of circumstances leading to a poor deployment last build cycle, I campaigned in our office to perform all future deployments with a dedicated build machine, and my boss accepted this proposal.

However, instead of getting an actual machine in our office to use, we're having to share a single machine with several other groups - and the hassle of having to leave my office with all the necessary information and then walk down a flight of stairs to another office just to perform a simple build is making me wonder why I ever proposed this in the first place.

The idea of having a separate build machine was, originally, to separate my own locally-written code from the code of several other developers, and to separate any hijacked files I had on my machine from deployment. It was also to resolve a growing concern I've had with our ClearCase file management system, which often refuses to let me deploy certain build activities unless I've also included another activity for which it 'has dependencies'.

Now that I'm actually going forward with this process, I'm wondering if I misunderstood the entire purpose of using a build machine - and since we're only using this machine for code deployment to our Test, Staging and Production environments, and not for our personal Developer testing deployments, I'm not sure it serves any purpose at all.

So, what is the actual reason for using a build machine, and have I even come close to using it correctly?

  • 166
    "the hassle of having to leave my office with all the necessary information and then walk down a flight of stairs to another office just to perform a simple build [...]" What do you mean exactly? You physically access that machine to create a build? – Vincent Savard Mar 3 '17 at 14:07
  • 6
    related: Responsibilities of Build Script and Build Server – gnat Mar 3 '17 at 14:08
  • 13
    The real WTF is Clearcase, which is worse than all the modern open source alternatives. What language(s) is this project and how large/complex is the build? – pjc50 Mar 4 '17 at 0:38
  • 7
    Are you using tools? Git/SVN and Jenkins/Team City/Octopus/TFS, etc? Or are you just logging into another computer, loading Visual Studio or what have you... copying project, loading, compiling... Are you using professional tools or doing it manually? – WernerCD Mar 4 '17 at 0:58
  • 84
    My build machine is somewhere in South Carolina and I am in Seattle. I assure you I do not walk down any set of stairs to use it. I think the last time I had physical access to a build machine was when I was the intern in charge of build machines for Microsoft compilers in 1994, when they fit into a small closet; they are now an entire data center somewhere. Get the machine on your network; better yet, get it in the cloud and make someone else take care of it. – Eric Lippert Mar 4 '17 at 8:07
137

Normally you wouldn't just have a dedicated build machine, but also run a build server on that dedicated machine. A dedicated build machine merely offers the advantage of never blocking the work of a developer and deploying from a centralized machine.

A build server offers much more. A build sever allows for CI (continuous integration), meaning that it will automatically build on every push to your VCS (like git), might even execute unit tests if you have them and allows for "one click deployment". Build servers can notify you per mail if builds or tests fail. They offer historic data and trends on what happened.

Build servers can generally be accessed by multiple users or teams at once, by using a web gui that runs in a browser.

In the Java world one of the most used build servers is Jenkins. Jenkins works perfectly fine with C++ builds as well (Since you seem to use those two languages). Jenkins calls itself automation server, since it can run all kinds of tasks that don't have to be related to programming and building.

  • 3
    I actually haven't touched c++ since college, but I can understand the usefulness. Though in this case, I think what we're actually doing might be far removed from the intended purpose. – Zibbobz Mar 3 '17 at 14:22
  • 21
    For some languages (C++ in particular) having a dedicated box with more processing power can be useful, too, given it's a relatively slow compilation. – enderland Mar 3 '17 at 15:16
  • 31
    Continuous Integration means more than just having a build server -- it also means everyone integrates each others changes as frequently as possible, to avoid "big bang" merge problems. Otherwise, spot-on. – Rob Crawford Mar 3 '17 at 18:00
  • While I like the power that an example gives to the point being made here, I still think the last paragraph is totally unnecessary in this answer. – Pierre Arlaud Mar 4 '17 at 15:09
  • 3
    "A dedicated build machine merely offers the advantage of never blocking the work of a developer and deploying from a centralized machine." Is not totally true. The asker pointed out that it is very easy to have an unclean environment on a developers machine. Included libraries and other environment variables can all change the build result. A dedicated machine should have a well documented environment to allow for easy build recreation. – TafT Mar 6 '17 at 16:45
107

In addition to Traubenfuchs' answer, you have hinted at another reason for a build machine in your question.

Just because the software builds on your machine, it doesn't mean that it will build on anyone else's. You may be relying on some random files that just happen to be on your machine (and may not even be under version control). You may be relying on some forgotten application or library that is called up from an obscure build script.

If you have a dedicated build machine, you should know what is installed on it. This should be well documented. If there is ever a need to rebuild the software, perhaps years later, it should only be necessary to create a new build machine with the documented stuff installed on it.

  • 37
    + 1 The amount of times something works on a developer's machine, and not the rest of his team's... – user2259716 Mar 3 '17 at 15:31
  • 45
    This. The main purpose of building on another machine is to have reproducible builds; notably by eliminating non-committed stuff, disparate environment variables, etc... from the equation. – Matthieu M. Mar 3 '17 at 16:01
  • 9
    Extend on that: use a new, clean container image from which you start each build. Then have a bootstrap script setup the system. That truly guarantees reproducible builds. – Matthias Kuhn Mar 3 '17 at 20:12
  • 9
    @MatthiasKuhn: truly (byte for byte) reproducible builds require much more, cf: reproducible-builds.org wiki.debian.org/ReproducibleBuilds – ninjalj Mar 5 '17 at 12:04
  • 1
    Also, just because the Linux version of the product builds and passes tests on your Linux desktop does not mean that the Windows version or the Mac version will build. – Solomon Slow Mar 6 '17 at 21:59
53

The main reason for having a dedicated build machine is to get consistent builds regardless of who is doing the build. Developer workstations are rarely (read: never) identical. It's hard to know that each build is using the same exact versions of dependencies and compilers etc. One of the worst problems with dev workstation builds is that developers can build from code that is not checked into version control.

It's not clear what platform/language(s) you are using but you ideally should have a build server that pulls directly from source control. That is, when a build is required, it will retrieve the source from a given version of the repository and compile it automatically. This requires the use of automated build tools to script the build. If you don't have this, that should be step #1.

Keep in mind that there is nothing wrong with building locally for development. You should definitely being working locally run unit tests, code quality analysis and for honing build scripts. Otherwise you will be wasting a lot of time. The output of the build server is for anything that you want to potentially move to production. All QA activities such as integration and acceptance testing should be done only with builds from the build server.

  • In addition, additional virus resistance. – Joshua Mar 4 '17 at 15:07
  • @Joshua What gives you that idea? – jpmc26 Mar 5 '17 at 6:08
  • 14
    @jpmc26: The build machine gets far less software installed on it, needs less remote access points, and nobody is opening web browsers to random internet sites on it. – Joshua Mar 5 '17 at 16:16
19

The other answers quite correctly noted that you should automate the build, which means it isn't necessary to walk to another office. However, let me propose a certain number of steps you could take to improve your build process:

  • Firstly, set up remote access to the build server! If you build manually by typing the command "make", this means you don't have to walk to another office anymore to type "make", you can just SSH into the build server and type "make". If you don't use make or a similar build system yet, take such a build system into use.
  • Secondly, install a continuous integration environment that automatically pulls latest changes from the version control system (you have a version control system, right? if not, that would be an additional step) and builds them. I recommend Jenkins. Set up Jenkins to run your unit tests and system-level integration tests as well (you do have both, right? if not, create them starting from unit tests and then ending into system-level integration tests).
  • Thirdly, if you find sharing the same machine with other teams problematic (such as if you have different opinions on what operating system and which version and bittiness you should use), consider using virtualization. A good server today can run a huge number of virtual machines. Perhaps you could set yourself up a 32-bit machine and a 64-bit machine so that you know the build works on both architectures.
  • Lastly, this may not be necessary: if you absolutely must have a dedicated machine, such as if your application performance is of great importance and other builds / test runs running at the same time affect your results too much, install a dedicated hardware server than only you use. However, on recent servers that may have as many as 40 virtual CPU cores or even more, it is relatively straightforward to create a few virtual machines that don't share access to the same CPU cores.

I would consider a shared machine much better than manual builds. My current project uses a virtual machine now, but due to the need for system-level integration performance tests, we're moving to a dedicated server with 40 virtual CPU cores of which the performance tests requires 17.

16

... instead of getting an actual machine in our office to use, we're having to share a single machine with several other groups ...

You say that like it's a bad thing.

You now have a common build server that all of your builds - yours and the other teams' - are built through. Consistency of build? Check.

... the hassle of having to leave my office with all the necessary information and then walk down a flight of stairs to another office just to perform a simple build is making me wonder why I ever proposed this in the first place.

You're still performing the build manually and that's not good.

You need a server process to which you submit/ queue requests for builds to be done on your behalf and have that process send you back the result(s).

  • 5
    "You say that like it's a bad thing." It could be if they have free reign to install whatever junk they want. If they're remoting in or physically accessing it, I don't see how that could be prevented. – jpmc26 Mar 3 '17 at 21:46
  • 7
    "You say that like it's a bad thing. You now have a common build server that all of your builds - yours and the other teams' - are built through. Consistency of build? Check". That's the complete opposite of a consistent build! If any other team decides to update their compiler or any other tool you're suddenly dealing with a completely different build environment. That's pretty much a worst case scenario (apart from not having a build machine to begin with). – Voo Mar 4 '17 at 0:20
  • 1
    Agreed on wanting to have a automatic process for creating new builds, but at the same time you also want to virtualize your build agents to make sure your build environment is under your own control. VMs are an amazing tool for dev ops and you should make the most of it. – Voo Mar 4 '17 at 0:23
  • @Voo It's hardly a worst case scenario, it's a middle case scenario. What the asker currently has is a worst case scenario. (Also note that if you never reproduce builds, random compiler upgrades aren't too much of a problem) – immibis Mar 7 '17 at 7:37
  • @immibis You did read the part in parens right after the part you quoted? It is a worst case scenario if a build server is involved. The problem with non reproducible bugs is that you can't really do Hotfixes if your whole build environment changed in the meantime. That isn't much of a problem if you only have a single released version that's kept close to trunk, but in all other cases it's pretty bad. – Voo Mar 7 '17 at 8:43
1

In addition to other relevant answers, it also sounds like you're running your builds directly on the machine in question.

For a reliable build system, especially when sharing the build machine with other users, it's normal to run your builds within a virtual machine. This ensures other users cannot change the behaviour of your builds by installing their own versions of applications or libraries which your code depends on. A great advantage of this is that the VM can easily be backed up, and it can also easily be cloned to any other PC (including your own development machine).

1

It provides a centralized, neutral location to perform builds, independent of the IDE, OS, library configurations of individual developers.

With a dedicated build machine, you can have it rebuild every time there is a code push to the repository. When someone breaks the build, the process can immediately send out an alert so that the problem can be rectified immediately.

In addition to making everything more repeatable and reliable and ensuring the repository isn't full of broken garbage with lurking dependency issues, it makes the lives of developers easier because all they have to do to get the build working on their machine is to copy whatever is being done on the build machine.

  • 4
    this doesn't seem to offer anything substantial over points made and explained in prior 6 answers – gnat Mar 6 '17 at 16:46

protected by gnat Mar 6 '17 at 16:46

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.