How to cope with the problem of (compiling) a large code base?

Question

Although I can code, I don't yet have any experience with working on large projects. What I did so far was either coding small programs that get compiled in matter of seconds (various c/c++ exercises like algorithms, programming principles, ideas, paradigms, or just trying out api's...) or working on some smaller projects that were made in a scripting language(s) (python, php, js) where no compiling is needed.

The thing is, when coding in a scripting language, whenever I want to try if something works - I just run the script and see what happens. If things don't work, I can simply change the code and try it out again by running the script again and keep on doing that until I get the result that I wanted.. My point is that you don't have to wait for anything to compile and because of that it is quite easy to take a big code base, modify it, add something to it, or simply play with it - you can see the changes instantly.

As an example I will take Wordpress. It is quite easy to try and figure it out how to create a plugin for it. First you start by creating a simple "Hello World" plugin, then you make a simple interface for admin panel to familiarize yourself with API, then you build it up and make something more complex, in the mean time changing how it looks a couple of times.. The idea of having to recompile something as big as WP over and over again, after each minor change to try "if it works" and "how it work/feels" just seems inefficient, slow and wrong.

Now, how could I do that with a project that is written in a compiled language? I would like to contribute to some open-source projects and this question keeps bugging me. The situation probably differs from project to project where some of them that were pre thought wisely will be "modular" in some way while others will just be one big blob that needs to be recompiled again and again.

I would like to know more about how this is done properly. What are some common practices, approaches and project designs (patterns?) to cope with this? How is this "modularity" called in programmers world and what should I google for to learn more about this? Is it often that projects grow out of their first thought proportions which becomes troublesome after a while? Is there any way to avoid long compiling of not-so-well designed projects? A way to somehow modularize them (maybe excluding non-vital parts of program while developing (any other ideas?))?

Thanks.

Answer 1

Just like it has been said, you never recompile the whole project each time you make a small change. Instead you only recompile the part of the code that has changed, as well as all code depending on it.

In C/C++, compiling is pretty straightforward. You compile translate each source file into machine code (we call them object files *.o) and then you link all your object files into one big executable.

Just like MainMa mentioned, some libraries are built into separate files, that will be linked dynamically at run-time with the executable. These libraries are called Shared Objects (*.so) in Unix and Dynamically Linked Libraries (DLL) in Windows. Dynamic libraries have many advantages, one of which being that you don't need to compile/link them, unless their source code effectively change.

There are build automation tools that help you:

• Specify dependencies between different parts of your source tree.
• Launch punctual, discreet compilations only in the part that was modified.

The most famous ones (make, ant, maven,...) can detect automatically which parts of the code has been changed since last compile, and exactly what object/binary needs to be updated.

However, this comes a the (relatively small) cost of having to write a "build script". It is a file containing all the information about your build, like defining the targets and their dependencies, defining which compiler you want and which options to use, defining your build environment, your library paths, ... You maybe heard about Makefiles (very common in the Unix world), or build.xml (very popular in the Java world). This is what they do.

Answer 2

You don't recompile the whole project every time. For example, if it's a C/C++ application, there are chances it will be separated into libraries (DLLs in Windows), every library being compiled separately.

The project itself is generally compiled daily on a dedicated server: those are nightly builds. This process can take a large amount of time, because it included not only the compile time, but also the time spent running unit tests, other tests and other processes.

Answer 3

I think what all of the answers so far have been alluding too, is that large software projects are almost always broken down into much smaller pieces. Each piece is normally stored in it's own file.

These pieces are individually compiled to create objects. The objects are then linked together to form the final product. [In a way, it is sort of like building stuff out of Legos. You don't try to mold the final thing out of one big piece of plastic, instead you combine a bunch of smaller pieces to make it.]

Breaking the project into pieces that are individually compiled allows some neat things to happen.

Incremental Building

First off, when you change one piece, you usually don't have to recompile all of the pieces. Generally speaking, as long as you don't change how other pieces interact with your piece, the others do not need to be recompiled.

This gives rise to the idea of incremental building. When doing an incremental build, only the pieces of that were affected by the change are recompiled. This greatly speeds up development time. True, you may still have to wait for everything to be relinked, but that it still a savings over having to both recompile and relink everything. (BTW: Some systems/languages do support incremental linking so that only the things that changed have to be relinked. The cost for this usually is in poor code performance and size.)

Unit Testing

The second thing that having small pieces allows you to do is look at individually testing the pieces before they are combined. This is known as Unit Testing. In Unit Testing, each unit is individually tested before it is integrated (combined) with the rest of the system. Unit tests are normally written so that they can be quickly run without involving the rest of the system.

The limiting case of applying testing is seen in Test Driven Development (TDD). In this development model, no code is written/modified unless it is to fix a failed test.

Making it Easier

So breaking down things seems good, but it also seems like a lot of work is needed to build the project: you need to figure our what pieces changed and what depends on those pieces, compile each piece, and then link everything together.

Luckily, programmers are lazy*, so they invent a lot of tools to make their jobs easier. To that end, many tools have been written to automate the task above. The most famous of these have already been mentioned (make, ant, maven). These tools allow you to define what pieces need to be put together to make your final project and how the pieces depend on each other (i.e. if you change this, this needs to be recompiled). The result is that issuing just one command does figures out what needs to be recompiled, compiles it, and relinks everything.

But that still leaves figuring out how things relate to each other. That's a lot of work and as I said before, programmers are lazy. So they have come up with another class of tools. These tools have been written to determine the dependencies for you! Often the tools are part of Integrated Development Environments (IDEs) like Eclipse and Visual Studio, but there are also some standalone ones used for both generic and specific applications (makedep, QMake for Qt programs).

*Actually, programmers aren't really lazy, they just like to spend their time working on problems, not doing repetitive tasks that can be automated by a program.

Answer 4

Here's my list of stuff you can try to speed up C/C++ builds:

• Are you setup to only rebuild what has changed? Most environments do this by default. There's no need to recompile a file if it or none of the headers has changed. Similarly, there's no reason to rebuild a dll/exe if all the linked in objs/lib haven't changed.
• Put 3rd party stuff that never changes and the associated headers in some read only code library area. You only need the headers and the associated binaries. You should never need to rebuild this from source other than maybe once.
• When rebuilding everything, the two limiting factors in my experience have been number of cores and disk speed. Get a beefy quad core, hyperthreaded machine with a really good hdd and your performance will improve. Consider a solid state drive -- keep in mind the cheap ones may be worse than a good hdd. Consider using raid to increase your hdd
• Use a distributed build system such as Incredibuild which will split up compilation across other work stations on your network. (Make sure you have a solid network).
• Setup a unity build to save you from constantly reloading header files.

Answer 5

The idea of having to recompile something as big as WP over and over again, after each minor change to try "if it works" and "how it work/feels" just seems inefficient, slow and wrong.

Executing something interpreted is also very inefficient and slow, and (arguably) wrong. You're complaining about the time requirements on the dev's PC, but not compiling causes time requirements on the user's PC, which is arguably a lot worse.

More importantly, modern systems can do quite advanced incremental rebuilds and it's not common to recompile the whole thing for minor changes- compiled systems can include script components, especially common for things like UI.

Answer 6

• Partial Rebuild

If the project implements proper compilation dependency DAG then you can get away with only recompiling the object files that your change affects.

• Multiple Compilation Process

Also assuming a proper compilation dependency DAG, you can compile using multiple processes. One job per core/cpu is the norm.

• Executable tests

You can create multiple executables for testing that only link particular object files.

Answer 7

In addition to MainMa's answer, we've also just upgraded the machines we work on. One of the best purchases we've made was an SSD for when you can't help but to recompile the entire project.

Another suggestion would be to try a different compiler. Back in the day, we switch from Java's compiler to Jikes and now we've moved on to using the compiler bundled with Eclipse (don't know if it has a name) which takes better advantage of multicore processors.

Our 37,000 file project took around 15 minutes to compile from scratch before we made these changes. After changes it was cut down to 2-3 minutes.

Of course, its worth mentioning MainMa's point again. Don't recompile the entire project every time you want to see a change.