Build management: does this exist?

Question

I want something that I haven't so far been able to find in anything but theoretical papers and books. If it doesn't exist I'm rather dumbfounded because it sounds like a lot of people use these kinds of systems and that they work very well. Maybe what I need is some sort of combination of things, I'm not sure. Here are my requirements for the system:

• Can hook into SVN to be triggered via checkin's.
• Can configure independent components as project "pipelines"
  • I mean that a checkin to a component triggers that component to build, run unit tests, and spit out a distributable.
• Can configure components that depend on other components
  • If I check in to component A and component B depends on component A, component B has its A dependency updated and rebuilds upon completion of component A.
• Uses native build processes like msbuild or make.
  • I don't have to configure something totally new just for this.
• If a dependency fails the upstream components do not update it, but they will update other components that did not fail.

That's the build server end. I want developer side integration as well.

• A new developer should be able to run a command that gets everything and builds it all for the first time.
  • This would include all components for some given project.
• The same build, update, build process happens on a developer's machine.
• Use of any IDE

All of the above is automated. My team currently has this but does almost everything by hand :(

What I'm describing is described in many places including the book Continuous Delivery and the paper titled Continuous Release. The closest I seem to be able to find so far is Maven, but it doesn't seem to have the "can use native build system" feature (you tell it each file you want built and the compiler/linker....like an XML make file). It also seems very much geared toward Java and I want something language agnostic...in fact I want something that doesn't necessarily even build programs, it could be building anything that needs some form of compilation.

So, how? What? Where? and How much?

Answer 1

Maybe I'm totally mistaken, but I'm pretty sure you are looking for a build server. I've been using CruiseControl.Net for years which is a spin off of CruiseControl, and NAnt, a spin off of Ant. Since, CruiseControl can run NAnt scripts you can set up the build server to do pretty much anything. Though the two technologies are written in .Net, they are not limited to just compiling .Net projects.

Best of all they are both open source :D

EDIT: Sorry, forgot to mention that the same NAnt scripts can be run on your developers machines. The thing is, your projects 3rd party dependencies would need to be in source control as well as your code. I've always added a lib folder that contains all of my 3rd party libraries for your exact same situation. My thought is that I want ALL of my codes dependencies in source control so that all I need to do to build the software, even on a brand new machine, is checkout the code and compile, thats it!

Answer 2

Sounds like you want a concept like the old Borland Gauntlet. A key element of that was:

1. Checkins had to pass muster, or they would be flagged or reverted.
2. Automated builds would be done on checkin.

You could do a lot of the automated-build-server today, using just a plain vanilla installation of Jenkins [formerly Hudson]. It would check your subversion repository and check out the latest code, and try to build it, and if it passed, it would be flagged green, and if it failed, it would be flagged red.

As for some kind of automatic promotion system, I would never attempt such a thing in an automated way, as I feel it would be a kind of automatic-mess-maker. I would rely on the automated tool to "green light" branches of the code, and I would always keep a human "lead" or "integration engineer" or "code lieutenant" in the merge loop.

But if you really wanted some kind of an automatic push/merge/pull thing, and it was my job to build it for you, the next thing I would do is ditch Subversion and move to Mercurial or Git.

If you want something "language agnostic" but not platform agnostic, and which can do a lot more than any mere build-tool, then I would look into the Final Builder server. It was originally a Delphi build tool, but it now builds all kinds of stuff including Delphi, Visual C++, java and other stuff. It has a really neato "almost any kind of computer file or data movement process automation building system" that can build and assemble all kinds of stuff, not just software. In fact, it sounds like, the most advanced IT automation tool around.

Answer 3

Do "gated checking" in TFS sound useful? http://blogs.msdn.com/b/patcarna/archive/2009/06/29/an-introduction-to-gated-check-in.aspx

No? Ok. So, everything starts with a trigger. This can limit your choices unless you write some custom code.

This is a partial answer: """If I check in to component A and component B depends on component A, component B has its A dependency updated and rebuilds upon completion of component A.""" - your build tool, whatever it is must know how to do this. I know that VS2010 usually can.

What you described sounds like a pretty custom tool to me (and there are dozens of ways to build stuff, trust me. A web site would be very different from an embedded project).

So, my recommendation is to write EVERYTHING that is custom yourself. Of course, there are tools that can download the source code for you, a building tool, possibly a tool that emails results etc. But, what you need is custom. Therefore someone should spend 2 weeks upfront and just write it. And then also tweak it later.

Just make sure to use a laconic interpreted language, so that this tool can be easily tweaked by anyone. Python is a good one; PowerShell might also do.

I do not think you need XML config file if you will go this route; just do it in code. I have stored the settings in a separate .py file. You do not need a parser for that. You can just access that stuff directly.

I read a blog post somewhere which I cannot find now. It opined that XML as a tool sucks for build tool configuration, because when it comes to imperative, and not declarative logic that some tools cannot handle. "What is wrong with using code?" - the author wrote. I agree with him.

Where I work build scripts are done by someone whom I do not consider a great programmer, so our build system sucks, but politics makes it impossible for me to say so plainly. Hopefully you are not bound by the same chains.

Sometimes you have got to sacrifice a weekend and just get it done before a committee gets together and decides on some crappy "solution".

Answer 4

Yes it exists but thus far I have not found the perfect Swiss army knife for this...

For the server side I use Hudson / Jenkins. I personally would recommend Jenkins, it is a fork of Hudson following some political mishaps from Oracle after they took over the project from Sun.

In my current setup I had independent build scripts already in place for each project, putting all this in Hudson was a breeze and did not require any changes to my build scripts just some configuration tidbits on server side. Tweaking config further I was able to get it to do pretty much all that you mention + automatic generation of web portals for each project containing the most recent documentation, code metrics and test results. Builds are schedule to occur once a day (nightly build) where a more intricate test suite is run and triggered following commits with a lightweight test suits to give quick feedback.

On the client side I have not yet gotten as far as single click checkout and build but it's pretty close actually. I'm using a dependency management framework which allows very effective decoupling of libraries from each-other. It is thus possible for a dev to only get a single part of the whole yet still be able to build without hinderence as all dependencies are automatically pushed to a repository and made available.

Because of this the one button get everything became almost obsolete, that said, it would be possible to use that same dependency framework to perform this task and have it get all the information for me. For this I use Maven, not perfect but well worth the investment. Works wonders for Java setups however the dot net equivalents seems to be dragging behind somewhat.

Hope this helps