4

Like it happens to many, our C++ project grew larger and larger, and it finally hit the point where maintainability starts to be a concern, mostly due to build times: even by using ccache, every change requires anywhere from 30 seconds to 5 minutes to rebuild and test (actually it seems that "linking" is the time-consuming phase, but we are still investigating).

The project consists mainly of a big scientific library, along with some applications which make use of it. For example, there are applications to augment simulation models, or language bindings (python) for integration into other software packages.

Now, it would seem natural to split the library into smaller libraries: the code is already organized in "modules", so this would not be an issue. The problem is with dependency management: we can think of the whole project as a big DAG, where each node is a library module, or external library (due of the project nature, there are many external dependencies on other scientific libraries).

So ideally, when working on a module, we would like to rebuild only that module, plus maybe the related test app, and link it with its dependencies. This would speed up the development cycle considerably.

Consequently, the idea is to handle each "module" as an independent project, that will be separately built into a (static) library. I have two concerns with this approach:

  • Imagine that I am working on app A that depends on library B, which itself depends on library C. During development, I find that I need a new feature in library C, so I go ahead and code that. I now have to manually rebuild library C, then library B and finally library A before continuing with the development. When the dependency chain grows larger, this can be problematic.
  • If the dependencies of one module change, how to manage the impact on the whole DAG (other modules that change as a consequence)?

I have been researching into several tools that might assist with my scenario. Git submodules is one option, but it does not address my concerns above. Apache Ivy seems not suitable for what I am trying to achieve. I suspect that the answer involves some powerful build system (we are using qmake), so I looked into cmake (but I found its scripting language limited and hard to learn), and then buck and pants.

I am still unsure if this is the way to go, so I would be grateful if someone could share advice and/or experiences.

  • 1
    Any decent build system can handle dependencies between different modules/libraries. For qmake, you might find this SO post helpful: stackoverflow.com/questions/1538398/… – Doc Brown Nov 8 '17 at 15:34
  • 4
    @matpen - This question is a little confusing to me, as like Doc Brown is saying, this is something handled automatically by most build systems. For instance, we have about 50 projects containing around 3000 source files. It's buildable in Visual Studio (msbuild) and cmake. And if I change any particular source file, I rebuild the whole thing...Either build system handles rebuilding only what's necessary for me. Cmake is well worth the effort to learn, by the way. The idea is to avoid scripting. Ideally, you just give the build system information about your project. – zzxyz Nov 8 '17 at 18:21
  • 3
    "every change requires anywhere from 30 seconds to 5 minutes to rebuild and test" Haha! I wish header changes only took 5 minutes to rebuild on my main project. Running all unit tests takes around 45 minutes! (We usually let the CI machines run the unit tests for us.) – user1118321 Nov 9 '17 at 4:02
  • 1
    You don't explain what kind of project are you developing (web application, HPC numerical code, compiler), and you don't tell about its size (millions of source lines, thousands of files). What compiler, what buidl system? Do you use shared libraries? Perhaps you have too many files and most of them are too small. So edit your question to improve it. – Basile Starynkevitch Nov 9 '17 at 5:41
  • 1
    qmake will help you to create the makefiles you need for each lib, I guess. For starting the build processes in the correct order, you will need some hierarchical makefiles. I suggest you google for exactly this "hierarchical make" - you will find plenty of links. – Doc Brown Nov 9 '17 at 7:25
6

You mention static libraries in your question. I'm guessing (perhaps wrongly) that you code a program for Linux.

You should use shared libraries when possible. Then you need much less linking time (at the expense of runtime linking, which usually is fast enough, at start time).

During development, I find that I need a new feature in library C, so I go ahead and code that. I now have to manually rebuild library C, then library B and finally library A before continuing with the development.

This should not be the case with shared libraries (as long as their API did not change).

Consider perhaps some architecture using plugins, and which dynamically loads them (e.g. with dlopen on POSIX).

it seems that "linking" is the time-consuming phase

You need to be sure that is the case (e.g. by passing -time or even -ftime-report to g++). Then you might use the gold linker and/or visibility function attribute with g++.

Be sure that the APIs of your sub-modules or libraries don't change often.

Avoid too small C++ source files, e.g. prefer having ten C++ source files of two thousand lines each rather than a hundred C++ source files of only a two hundreds lines each. A C++ source file can define and implement several (related) functions or classes. Remember that C++ don't have modules, and that most standard headers (and your own ones) are quite big files (for example #include <vector> expands to about 10KLOC on my Linux desktop); what counts for compilation time is the size of preprocessed form, and template expansions need a lot of time too. See also this.

BTW, you also need to use a good build automation tool (such as ninja, GNU make - with parallelisation enabled thru make -j 8, etc...). You could also use distributed build tools (distcc, icecream, ...)

You might also (with GCC) consider using a precompiled header. Then, you'll probably want to have a single header file including all the other ones. You might want to use the PIMPL idiom.

See also this answer to a relevant question.


every change requires anywhere from 30 seconds to 5 minutes to rebuild and test

BTW, a five minute incremental build time seems rather small to me. Why are you complaining? I'm old enough to have had in the 1990s, on powerful Sun workstations of that time, build times of almost an hour (for a software of a few hundred thousands lines I wrote alone during several years). And I sometimes contribute to GCC itself even these years, whose rebuild time takes several hours.

  • +1 for distcc -- forgot the name of it when posting my answer! Didn't know gcc had precompiled header support which combined with gold's --incremental would probably reduce build times by an order of magnitude with very little effort. – Graeme Wicksted Nov 9 '17 at 14:38
1

Not sure what your environment/budget is but here are a few suggestions:

  • Visual Studio 2017 especially with /MP, precompiled headers, incremental builds, and possibly C++ modules
  • Incredibuild (never used personally)
  • Octobuild (never used personally)
  • Zapcc can be much faster than mainstream compilers

The usual advice applies for single and multiple projects: run the best CPU+RAM+SSD (or even ramdisk) you can buy, disable on-access virus scanning, and infrequently change headers where possible.

  • Precompiled Headers are an old trick going back to much slower machines. Given that a lot of code is in libraries and doesn't change the precompilation can save a lot of time without losing flexibility. – Michael Shopsin Nov 10 '17 at 20:51
1

There are already fine answers related to build systems and shared libs, but I'll tackle this from another angle I don't see people applying so often.

To me it's useful to separate stable code (as in unlikely to ever need further changes) from unstable code (code that naturally warrants further changes as the software expands). If you cannot do this and see the entirety of your codebase as potentially moving parts, even if the interfaces are completely stable, I think you are doing something wrong.

We already tend to do this naturally for open source third party libraries. We generally build them once, create a dylib ideally to avoid static linking times, and just look up symbols and call functions in them at runtime. We don't find a need to constantly rebuild them over and over since we have no interest in touching their source code, only using the available functionality -- basically build once and use forever. The code is 100% stable in that regard (there is no reason for it to ever change, at least not in our hands).

So for third parties I personally put them in a "third_parties" subdirectory and I only build that when I add a new third party library or replace an old version with a new version which is extremely rare, like once every few months minimum. I'm not rebuilding those over and over on a daily basis.

Same thing should be able to apply to your own codebase. I have a similar thing where I have a "libs" subdirectory with my own code and rarely ever build the resulting libraries since the code is very stable, reliable, efficient, tested thoroughly with unit tests, been through static analysis, and not something I need to change in the future. It's separated far away from the unstable code outside of the "libs" subdirectory which does need to change on a daily basis which does get rebuilt repeatedly.

So separating your codebase this way where stable parts you rarely, if ever, need to change from unstable parts which constantly warrant changes can be a useful way to organize your codebase -- not only in terms of optimizing build times but in order to better separate away stable packages from unstable packages with a goal to end up with as much stable code as possible that you can confidently say will not warrant any changes in any near future.

When you do that, those stable parts of your codebase can be built separately and away from the unstable parts using them. The unstable parts shouldn't take so long to build as your stable set of libraries expand and expand while the unstable parts shrink and shrink.

However, this tends to imply some code duplication. To be able to create a stable image library, it can't depend on, say, some unstable math library or else changes to the math library will warrant a rebuild if not further source changes to the image library. So I actually find it useful sometimes for, say, an image library to duplicate some math routines in order to become independent from any auxiliary math library. In that case, decoupling the code in this way through some modest duplication of logic can actually help make those packages a lot more stable, eliminating reasons for them to have to change and/or be rebuilt over and over. If the code is well-tested and works beautifully for years to come and might hardly ever have to be rebuilt, the modest duplication required for the library to achieve its independence and stability is hardly a problem.

Further it helps to achieve minimalism in your interfaces. If your image library aims to implement every single image operation ever imaginable to mankind, then its ambitions and monolithic nature will warrant neverending changes. It can never hope to become a stable library with such goals. If it only aims to provide basic image operations or even none at all with the ability to create image operations outside of the library using what the library provides, then it can potentially achieve a state of perfect stability (finding no reason to change in any near future).

So anyway, if you are going to start splitting up a codebase, I'd suggest to start off splitting the stable, well-tested parts where you're pretty sure you don't need to change them any further from the unstable parts where you at least anticipate some possibility of a need for future changes. Building the stable parts should invoke a separate build process (rarely applied) from the unstable parts (frequently rebuilt).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.