5

The project is using C++ and the code takes around half an hour to build on a 32 core box. That time can be longer, e.g. 1 hour+ on a developer's local machine.

I notice the efficiency is low when one is doing the build. Just sit and watch it is waste of time.

A typical case is: modify some code, then build, if build fails, modify again, build, test, if test fails, modify and build again.

Since the code is full of templates, so any modification in .H will take a long time to build.

If one developer wastes 2 hours daily in the build process, the cost would be too high.

How to improve the efficiency when dealing with a project that current code base needs a long time to build?

4
  • It seems quite astonishing that every small change either requires a complete rebuild, or that the build is badly configured. You can probably compile Windows 10 completely in less than that time.
    – Aganju
    Jun 30, 2019 at 3:16
  • template heavily used, header files only, change will take significant time for rebuild.
    – bugs king
    Jun 30, 2019 at 3:23
  • This depends a lot on what those templates are doing. Some uses of templates can be rewritten to use polymorphism, which trades run time against compilation time. That is especially useful at module boundaries, or if a non-templated helper can be extracted from a template. Explicit template instantiation can avoid part of template recompilation if the same template would otherwise be instantiated in multiple compilation units. Some cases of template fetishism are better resolved with the preprocessor.
    – amon
    Jun 30, 2019 at 6:58
  • 2
    First, get a copy of "Large Scale C++ Software Design". Then, Google for "reduce c++ buld times", you will find plenty of articles. Finally, reduce template usage in your team. Make it a rule to check in every code review if that the "extra run time speed" of a template solution (vs. a non-templated one) is really worth the reduced compile time speed you observe today.
    – Doc Brown
    Jun 30, 2019 at 7:58

4 Answers 4

5

One thing would be to reevaluate how much you are relying on templates. In typical C++ projects, creating your own templates is relatively rare.

Next thing is to make sure you are generating dependency files, so you can actually do incremental builds. This is surprisingly easy to get wrong in a lot of toolchains.

Next thing is to look into precompiled headers.

Aside from that, the main thing is just not compile the entire build every time. You are typically working in one file at a time. Just compile that one file, even if you have to temporarily copy it outside your full build system to do so. Make stubs for your dependencies, and create a file to do unit tests. Work on that one file in isolation, with rapid compile cycles, then copy it back into your main build when you are ready for integration testing.

Another thing is to build as much testability into your full builds as possible. If you have to decide between two algorithms, write it both ways and create a way to choose between them using runtime configuration. If you're trying to find the source of a bug, make a build with a ton of logging. Learn to use a debugger so you can make small changes at runtime.

Another thing is to work on two tasks at once. Have something else you can pick up while your first task is compiling.

9
  • I am talking about incremental build. The project does split into modules. But many modules are template header files. So if one change something in the header, it needs to build the module and the application that depends on it. The second, templates are heavily used, so many impl are in the headers as well.
    – bugs king
    Jun 30, 2019 at 5:01
  • 1
    "In typical C++ projects, creating your own templates is relatively rare" - it should be rare, unfortunately, today it is often not. In typical C++ projects, if there is not a highly experienced team manager who restricts usage of templates to the cases where they really make sense, medium-experiended guys often "fall in love" with over-templating code. The impact on build-times is often overlooked, or avoidance of templates is classified as a form of premature optimization.
    – Doc Brown
    Jun 30, 2019 at 7:51
  • 1
    The most concerning thing I'm hearing here isn't that they use templates. It's that they keep changing the headers. You need the interfaces between the modules to be more stable than the modules or the modules aren't going to do you any good. Spend some time getting your h files right so they'll stop changing so much. Jun 30, 2019 at 12:19
  • @candied_orange: the issue here is, what you suggest is usually mutual exclusive to the usage of templates in C++-
    – Doc Brown
    Jun 30, 2019 at 13:31
  • @DocBrown well it shouldn't be. If you want your code to be widely used then you have to act like it's widely used and make something stable that can be relied on. You can't keep stirring the interface. Jun 30, 2019 at 14:05
3

Additions to other answers which already suggested making proper incremental builds:

  • explore opportunity of distributed builds, with distcc for example.
  • split up headers to smaller pieces, so that changing some interface would not cause recompilation of functions which use some unrelated one in same header file
  • use forward declarations whenever possible. For example, if you have function or method Foo getFoo(Bar bar) you don't need implementation of Foo or Bar unless you actually call the getFoo(...). Note that you cannot forward declare a STL template.
  • limit use of templates to implementations, do not use them in interfaces
  • use pimpls
  • have CI which builds for any feature branch
  • do not make developers reset forth and back in history. Fetch master once, build it, then build only incrementally during development. Cherry-pick to feature branch using dedicated worktree, push and start new feature in master without. CI will let you know the unlikely event of compilation failure due to lack of synchronization.
  • during prototyping, use dirty tricks to not touch heavily used header files: unsafe casting, global variables, copying definitions to implementation files instead of including them. Of course this should be done properly once you settled with implementation

PS:

A typical case is: modify some code, then build, if build fails, modify again, build, test, if test fails, modify and build again.

Since the code is full of templates, so any modification in .H will take a long time to build.

you don't have to rebuild the whole project to find error in the template change in header. It should be possible to request compilation just one or several files in your project, and if they succeed most probably the header is correct.

2

Break up the code base into modules. Rebuild them only when changes impact them. Break up your headers into small groups of related functions so unused functions rarely get included.

Stable headers are key. Test them thoroughly before tossing them into the larger program.

Seriously, this is why we invented modules in the first place. Only recompile what has changed and let the linker connect the rest.

Find out what your bottle neck actually is. It might be the CPU isn't what's holding you up.

6
  • This sounds great in theory, but if you have ever tried this in a larger piece of C++ code, than you will understand why this recommendation is way-too-simple to be of any practical usage.
    – Doc Brown
    Jun 30, 2019 at 13:35
  • @DocBrown has c++ changed that much? 9 years ago this was spot on advise. Jun 30, 2019 at 14:22
  • C++ hasn't change that much. Your answer is just IMHO oversimplifying the stated problem to a degree where it won't be of any practical help. To be fair, the OP question is giving me the impression they are underestimating the problem as well (and I voted to close as "too broad"). And the link you gave - the highest voted answer (not the accepted one) contains some good starters for really solving the problem.
    – Doc Brown
    Jun 30, 2019 at 15:30
  • ... note that answer does not follow the line "break up everything into modules" (not saying the latter statement would be wrong)..
    – Doc Brown
    Jun 30, 2019 at 15:54
  • 1
    Isn't the problem that there's a lot of implementation logic inside those heavily templated header files? And if so, how can you keep them stable? E. g. you touch something - anything - in the project, and it's likely to be inside a header file.
    – Headcrab
    Jul 1, 2019 at 1:16
1

Slow feedback loops, regardless of language/framework/platform, leads to inefficiency. People really aren't good at jumping from context to context and making good use of every minute of the day, so management needs to understand the cost of the build times in this project are a developer might get to try 6 changes a day instead of 20+ with 15 minute builds.

Once this is realized and faster feedback loops are prioritized, look at some other answers for how that might be achieved. It's not going to happen by itself as a side project if developers are being told to prioritize some new feature instead.

Not the answer you're looking for? Browse other questions tagged or ask your own question.