I am in need of some help with philosophy and design of a continuous integration setup.

Our current CI setup uses buildbot. When I started out designing it, I inherited (well, not strictly, as I was involved in its design a year earlier) a bespoke CI builder that was tailored to run the entire build at once, overnight. After a while, we decided that this was insufficient, and started exploring different CI frameworks, eventually choosing buildbot. One of my goals in transitioning to buildbot (besides getting to enjoy all the whiz-bang extras) was to overcome some of the inadequacies of our bespoke nightly builder.

Humor me for a moment, and let me explain what I have inherited. The codebase for my company is almost 150 unique c++ Windows applications, each of which has dependencies on one or more of a dozen internal libraries (and many on 3rd party libraries as well). Some of these libraries are interdependent, and have depending applications that (while they have nothing to do with each other) have to be built with the same build of that library. Half of these applications and libraries are considered "legacy" and unportable, and must be built with several distinct configurations of the IBM compiler (for which I have written unique subclasses of Compile), and the other half are built with visual studio. The code for each compiler is stored in two separate Visual SourceSafe repositories (which I am simply handling using a bunch of ShellCommands, as there is no support for VSS).

Our original nightly builder simply took down the source for everything, and built stuff in a certain order. There was no way to build only a single application, or pick a revision, or to group things. It would launched virtual machines to build a number of the applications. It wasn't very robust, it wasn't distributable. It wasn't terribly extensible. I wanted to be able to overcame all of these limitations in buildbot.

The way I did this originally was to create entries for each of the applications we wanted to build (all 150ish of them), then create triggered schedulers that could build various applications as groups, and then subsume those groups under an overall nightly build scheduler. These could run on dedicated slaves (no more virtual machine chicanery), and if I wanted I could simply add new slaves. Now, if we want to do a full build out of schedule, it's one click, but we can also build just one application should we so desire.

There are four weaknesses of this approach, however. One is our source tree's complex web of dependencies. In order to simplify config maintenace, all builders are generated from a large dictionary. The dependencies are retrieved and built in a not-terribly robust fashion (namely, keying off of certain things in my build-target dictionary). The second is that each build has between 15 and 21 build steps, which is hard to browse and look at in the web interface, and since there are around 150 columns, takes forever to load (think from 30 seconds to multiple minutes). Thirdly, we no longer have autodiscovery of build targets (although, as much as one of my coworkers harps on me about this, I don't see what it got us in the first place). Finally, aformentioned coworker likes to constantly bring up the fact that we can no longer perform a full build on our local machine (though I never saw what that got us, either, considering that it took three times as long as the distributed build; I think he is just paranoically phobic of ever breaking the build).

Now, moving to new development, we are starting to use g++ and subversion (not porting the old repository, mind you - just for the new stuff). Also, we are starting to do more unit testing ("more" might give the wrong picture... it's more like any), and integration testing (using python). I'm having a hard time figuring out how to fit these into my existing configuration.

So, where have I gone wrong philosophically here? How can I best proceed forward (with buildbot - it's the only piece of the puzzle I have license to work on) so that my configuration is actually maintainable? How do I address some of my design's weaknesses? What really works in terms of CI strategies for large, (possibly over-)complex codebases?


I thought I explained my problem, but obviously I wasn't clear enough. I am not looking for suggestions for changing CI platforms. It is not going to happen, and answers suggesting that will not get accepted. What I want to know is how other people manage complicated codebases using CI. I have a dozen squared different products, and I have dependencies scattered to the wind, and they're all different. THIS is what I want to know how to deal with.

  • I, too, would like to know the answer to this. Our environment isn't as complex as yours, but we've got dependencies of dependencies of dependencies (in the case of the setup project, it has dependencies four layers deep). I don't know if each project should be a CI project, or if I should just use the Visual Studio .sln file to take care of it for me so that I don't have to re-create the dependency tree for each project (and future project).
    – moswald
    Commented Jul 5, 2011 at 20:24
  • Not being able to build on your local machine makes your CI server mission critical to your business. You may want to rethink this. Commented Jul 9, 2016 at 13:37

4 Answers 4


Although I haven't faced a situation as bad as you describe, I have been maintaining a CI configuration with tens of components, between which there are some 'simple' dependencies. I hope that my approach may give you some hints with which to proceed. The problem is definitely not related only to the choice of CI server, but also to the overall build process and project structure.

I shall break down the problem into 2 parts: Building and CI.


For "Building" I means the process of, changing the source code on hand to the final artifact. Our company mainly uses Java in development, and the build tool we used is Maven. You will probably unable to use that due to nature of your project, but there are some valuable concept in Maven worth noting:

1) In Maven world, each artifact (a lib, the real program etc) needs to be clearly isolated and decoupled. Dependencies between artifact should be clear. I should emphasize, messy dependency, especially circular dependencies between your build artifacts are going to make the build process a mess.

For example, I saw some java projects before that, although after whole build process there are several JARs (you may consider it as lib/dll in Java) built, they are in fact inter-dependent. Like, A.jar is using things in B.jar, and vice versa. Such kind of 'modularization' is totally meaningless. A.jar and B.jar always needs to be deployed and used together. The implication is, later if you want to make them separated into different projects (for reuse in other projects for example), u will not be able to do so, coz you cannot determine which project, A or B, to build first.

Yes, it needs to be catered in your software's design, but I always believe that, instead of paying time to make a complicated building tools/model works in a messy project, I'd rather spend time to reorganize the design to make use a simple building model.

2) Dependencies should be declarative. There are lots of build process I saw before, which they includes all libs it needs locally in the project. It will make build process extremely troublesome if some of the libs are in fact other artifacts that you need to build.

3) "Centralized" storage for the artifacts for getting dependencies, or to deploy artifacts once it is compiled. It doesn't need to be "centralized" for whole office (it will be great if it is), just a local directory will already be fine.

More elaboration on 2 and 3. Just to give an example, I have come across a project that involved 3 independent projects. The each project is built base on the source, plus the libs under the the lib/ directory in the source directory. Project A will build several libs, which is in turn used by project B and C. There are quite some drawbacks: build procedure is complicated and harder to automate; the source control is getting bloated with unnecessary duplicated JARs reused in different project

In Maven's world, what it is done is, project B and C do not really contains A.jar (and other dependencies, like other 3rd party lib) in the project source. It is declarative. For example, in the build configuration of project B, it simply declare that it need: A.lib v1.0, xyz.lib v2.1 etc, and the build script will look up the lib from /A/1.0/A.jar and /xyz/2.1/xyz.lib

For non-Maven world, this artifact dir just need to be one or two directory with agreed directory structure. You may put all 3rd party libs to a share location and let developers sync or copy to their local machines. In my C++ projects I did many years before, what I am doing is setting the lib and header to ${artifact_dir}/lib_name/ver, and with artifact_id declared as environment variable.

When project A is built, it will have a copy of its result in that artifact_dir, so that when we build project B, B can get the result of A automatically, without manual copying.

4) Non mutable release. Once you release A.lib 1.0, that's it, you won't expect A.lib 1.0's content change after 1 month just bcos there are some bug fixes. In such case, it should be A.lib 1.1. The artifact of a changing code base should consider a special "version", for which in Maven we call it snapshot.

Non-mutable release is more an ethic issue. But what it solved is clear: When u have lots of projects, using the same lib, u know which version of that lib you are using, and you will be sure that, same version of lib used in different projects, are indeed the same. I think many of us have go thru the problem of: Why project X and Y is both using lib A but the build result is different? (and turns out that the lib A using by X and Y is in fact different after digging into the content or file size of the lib).

All these ensure that your projects can build independently, without much manual tricks like, build project A first, copy A.lib to project B, and then build project B...

After going through an exercise like this, for example, when you build project A, it will try to get the dependencies from the centralized artifact storage. In case some dependencies (which are other projects of your company, e.g. project B) is not found, what u need to do is, get the source of project B, build it (which will deploy to the centralized storage upon success), and then build project A again.


With a easy build system, with clear dependencies, CI will be much easier. I will expect your CI server serves the following requirement:

1) Monitor the source control, only checkout+build when there is changes in source

2) Able to setup dependencies between projects

With a clear dependency for projects, you simply need to setup projects in CI according to your actual projects, setup CI project dependency according to your projects' dependency. Your CI server should be set to build dependent projects before building any project (and of course, build only happens if the project source really have change).

If everything goes right, you should have a huge, complex, but still manageable CI (and even more important, a manageable project structure)

  • I like where this answer is going, but could you go into more detail in the 'building' section? That is, give some examples, explain some of the concepts more fully, elaborate.
    – Nate
    Commented Jul 6, 2011 at 13:12
  • hm... like which part? I try to edit the post to give more detail of each part. It will be better if u can tell me which part u feel need to be elaborated on. By the way, if you are using Java too, take a look in maven.apache.org. Maven may not be the easiest thing to adopt but it force you to make your things clean and organized. Commented Jul 7, 2011 at 7:23

What has worked in similar situations for me:

  • App-level abstraction by shunting some parts of the build into dedicated build apps (in our case these are PHP scripts called from the main build). This can reduce a few dozen lines of build steps into one build step.
  • Build-level abstraction by making sub-build-scripts that get launched from the main build script. No idea whether it's relevant for buildbot (no experience with that product).

I would suggest that you treat the build itself as a software development project. This means that you need to modularize the build codebase and write some automated tests for it (e.g. build a known revision number and check whether it produces correct results).


I would consider Jenkins as a CI server, you can look at the features here:


Why? It's easy to install, it has a wonderful interface, it's easy to configure and extend and there are A LOT of ready plugins for almost everything:


Give it a try :)

  • 2
    I sense that you read the title to my question, but did not read the body... I'm not looking for a product suggestion. I have chosen a product. I am looking for how to organize a complex CI setup.
    – Nate
    Commented Jul 2, 2011 at 16:19
  • Nate, I read all your post and I think you have choosen the wrong product, that's why I suggested Jenkins. I use Jenkins at work for various kind of testing of C++ products (even tests written in multiple languages) with clustering on multiple machines and it's running very well. It's very easy to administer and configure. Really, try it :) Commented Jul 2, 2011 at 16:51
  • Look at this blog post too: vperic.blogspot.com/2011/05/… Commented Jul 2, 2011 at 17:00

We now use Go by ThoughtWorks prior to that, we were using CruiseControl.Net. It has been useful considering we have two teams half the world away from each other and there's a big time zone difference between us.

We are managing multiple projects and most of the tasks involves overlaps between two developers in both sides of the globe, so we have to keep in mind that all files should not break somebody else's build.

Also management is easier with Go and being an Agile Process diehard, it has given us lesser headaches after we installed it. Testing has also been integrated with Go.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.