How can I prevent header hell?

Question

We are starting a new project, from scratch. About eight developers, a dozen or so subsystems, each with four or five source files.

What can we do to prevent “header hell”, AKA “spaghetti headers”?

• One header per source file?
• Plus one per subsystem?
• Separate typdefs, stucts & enums from function prototypes?
• Separate subsystem internal from subsystem external stuff?
• Insist that every single file, whether header or source must be standalone compilable?

I am not asking for a “best” way, just pointer as to what to watch out for and what could cause grief, so that we can try to avoid it.

This will be a C++ project, but C info would help future readers.

Answer 1

Simple method: One header per source file. If you have a complete subsystem where users are not expected to know about the source files, have one header for the subsystem including all required header files.

Any header file should be compilable on its own (or let's say a source file including any single header should compile). It's a pain if I found which header file contains what I want, and then I have to hunt down the other header files. A simple way to enforce this is to have every source file include it's header file first (thanks doug65536, I think I do that most of the time without even realising).

Make sure you use available tools to keep compile times down - each header must be included only once, use pre-compiled headers to keep compile times down, use pre-compiled modules if possible to keep compile times further down.

Answer 2

By far the most important requirement is to reduce dependencies between your source files. In C++, it is common to use one source file and one header per class. Therefore, if you have a good class design, you will not even come close to header hell.

You can also view this the other way round: if you already have header hell in your project, you can be quite sure that the software design needs to be improved.

To answer your specific questions:

• One header per source file? → Yes, this works well in most cases and makes it easier to find stuff. But don't make it a religion.
• Plus one per subsystem? → No, why would you want do do this?
• Separate typedefs, structs & enums from function prototypes? → No, functions and related types belong together.
• Separate subsystem internal from subsystem external stuff? → Yes, of course. This will reduce the amount of dependencies.
• Insist that every single file, whether header or source by standalones compilable? → Yes, never require that any header has to be included before another header.

Answer 3

In addition to the other recommendations, along the lines of reducing dependencies (mainly applicable to C++):

1. Only include what you really need, where you need it (lowest level possible). E. g. don't include in a header if you need the calls only in the source.
2. Use forward declarations in headers wherever possible (header contains only pointers or references to other classes).
3. Clean up the includes after each refactoring (comment them out, see where compilation fails, move them there, remove the still commented include lines).
4. Don't pack too much common facilities into the same file; split them by functionality (e.g. Logger is one class, thus one header and one source file; SystemHelper dito. etc.).
5. Stick to OO principles, even if all you get is a class consisting solely of static methods (instead of standalone functions) - or use a namespace instead.
6. For certain common facilities the singleton pattern is rather useful, as you don't need to request the instance from some other, unrelated object.

Answer 4

One header per source file, which defines what its source file implements/exports.

As many header files as necessary, included in each source file (starting with its own header).

Avoid including (minimize the inclusion of) header files within other header files (to avoid circular dependencies). For details see this answer to "can two classes see each other using C++?"

There's a whole book on this subject, Large-Scale C++ Software Design by Lakos. It describes having "layers" of software: high-level layers use lower-level layers not vice versa, which again avoids circular dependencies.

Answer 5

I would claim your question is fundamentally unanswerable, since there are two kinds of header hell:

• The kind where you need to include a million different headers, and who in hell can even remember all of them? And maintain those lists of headers? Ugh.
• The kind where you include one thing, and find out you've included the whole Tower of Babel (or should I say tower of Boost?...)

the thing is, if you try to avoid the former you end up, to some extent, with the latter, and vice-versa.

There's also a third kind of hell, which is circular dependencies. These might pop up if you're not careful... avoiding them is not super-complicated, but you do need to take the time to think about how to do it. See John Lakos talk on Levelization in CppCon 2016 (or just the slides).

Answer 6

Decoupling

It's ultimately about decoupling to me at the end of the day at the most fundamental design level devoid of the nuance of the characteristics of our compilers and linkers. I mean you can do things like make each header define just one class, use pimpls, forward declarations to types that only need be declared, not defined, maybe even use headers that just contain forward declarations (ex: <iosfwd>), one header per source file, organize the system consistently based on type of thing being declared/defined, etc.

Techniques to Reduce "Compile-Time Dependencies"

And some of the techniques can help quite a bit but you can find yourself exhausting these practices and still find your average source file in your system needs a two-page preamble of #include directives to do anything slightly meaningful with skyrocketed build times if you place too much focus on reducing compile-time dependencies at the header level without reducing logical dependencies in your interface designs, and while that might not be considered "spaghetti headers" strictly speaking, I'd still say it translates to similar detrimental issues to productivity in practice. At the end of the day if your compilation units still require a boatload of information to be visible to do anything, then it's going to translate to increasing build times and multiply the reasons you have to potentially go back and have to change things while making developers feel like they're headbutting the system just trying to get their daily coding finished. It's like those types of techniques absent proper decoupling might hand you one spaghetti noodle at a time but you're still going to end up making spaghetti in your compilation units anyway.

You can, for example, make each subsystem provide one very abstract header file and interface. But if the subsystems aren't decoupled from one another, then you get something resembling spaghetti again with subsystem interfaces depending on other subsystem interfaces with a dependency graph that looks like a mess in order to work.

Forward Declarations to External Types

Of all the techniques I exhausted to try to get a former codebase that took two hours to build while the developers sometimes waited 2 days for their turn at CI on our build servers (you can almost imagine those build machines as exhausted beasts of burden frantically trying to keep up and failing while developers push their changes), the most questionable to me was forward declaring types defined in other headers. And I did manage to get that codebase down to 40 mins or so after ages of doing this in little incremental steps while trying to reduce "header spaghetti", the most questionable practice in hindsight (as in making me lose sight of the fundamental nature of design while tunnel visioned on header interdependencies) was forward declaring types defined in other headers.

If you imagine a Foo.hpp header that has something like:

#include "Bar.hpp"

And it only uses Bar in the header a way that requires it declaration, not definition. then it might seem like a no-brainer to declare class Bar; to avoid making the definition of Bar visible in the header. Except in practice often you'll either find most of the compilation units that use Foo.hpp still end up needing Bar to be defined anyway with the additional burden of having to include Bar.hpp themselves on top of Foo.hpp, or you run into another scenario where that does genuinely help and 99% of your compilation units can work without including Bar.hpp, except then it raises the more fundamental design question (or at least I think it should nowadays) of why they need to even see the declaration of Bar and why Foo even needs to be bothered to know about it if it's irrelevant to most use cases (why burden a design with dependencies to another one barely ever used?).

Because conceptually we haven't really decoupled Foo from Bar. We've just made it so the header of Foo doesn't need as much info about the header of Bar, and that's not nearly as substantial as a design which genuinely makes these two completely independent of each other.

Embedded Scripting

This is really for larger scale codebases but another technique I find immensely useful is to use an embedded scripting language for at least the most high-level parts of your system. I found I was able to embed Lua in a day and have it uniformly able to call all commands in our system (the commands were abstract, thankfully). Unfortunately I ran into a roadblock where the devs distrusted the introduction of another language and, perhaps most bizarrely, with performance as their greatest suspicion. Yet while I could understand other concerns, performance should be a non-issue if we're only utilizing the script to invoke commands when users click buttons, for example, that perform no hefty loops of their own (what are we trying to do, worry about nanosecond differences in response times for a button click?). So that might not be so applicable to you but it's an option I'd consider for larger codebases focusing more on the issues of build times than the other issues associated with "header spaghetti", since it can eliminate a lot of what might otherwise need to be pre-compiled and linked (nowadays I write even all my integration tests in high-level script which is a load off on the build system and CI).

Example

Meanwhile the most effective way I've ever witnessed after exhausting techniques to reduce compile times in large codebases are architectures which genuinely reduce the amount of information required for any one thing in the system to work, not just decoupling one header from another from a compiler perspective but requiring the users of these interfaces to do what they need to do while knowing (both from a human and compiler standpoint, true decoupling that goes beyond compiler dependencies) the bare minimum.

The ECS is just one example (and I'm not suggesting you use one), but encountering it showed me that you can have some really epic codebases that still build surprisingly quickly while happily utilizing templates and lots of other goodies because the ECS, by nature, creates a very decoupled architecture where systems only need to know about the ECS database and typically only a handful of component types (sometimes just one) to do their thing:

Design, Design, Design

And these sorts of decoupled architectural designs at a human, conceptual level is more effective in terms of minimizing compile-times than any of the techniques I explored above as your codebase grows and grows and grows, because that growth doesn't translate to your average compilation unit multiplying the amount information required at compilation and link times to work (any system which requires your average developer to include a boatload of stuff to do anything also requires them and not just the compiler to know about a great deal of information to do anything). It also has more benefits than reduced build times and untangling headers, since it also means your developers don't need to know much about the system beyond what is immediately required in order to do something with it.

If, for example, you can hire an expert physics developer to develop a physics engine for your AAA game which spans millions of LOC, and he can get started very quickly while knowing the absolute bare minimum information as far as things like types and interfaces available as well as your system concepts, then that's naturally going to translate to reduced amount of information for both him and the compiler to require to build his physics engine, and likewise translate to a great reduction in build times while generally implying that there's nothing resembling spaghetti anywhere in the system. And that's what I'm suggesting to prioritize above all these other techniques: how you design your systems. Exhausting other techniques will be icing on the top if you do it while, otherwise, it can be like icing without the cake in the worst-case scenario.

Answer 7

It is a matter of opinion. See this answer and that one. And it also depends a lot on the project size (if you believe you'll have millions of source lines in your project, it is not the same as having a few dozen thousands of them).

In contrast to other answers, I recommend one (rather large) public header per sub-system (which could include "private" headers, perhaps having separate files for implementations of many inlined functions). You could even consider a header having only several #include directive.

I don't think that many header files is recommended. In particular, I don't recommend one header file per class, or many small header files of a few dozen lines each.

^{(If you have a big lot of small files, you'll need to include a lot of them in every small translation unit, and the overall build time might suffer)}

What you really want is to identify, for each sub-system and file, the main developer responsible for it.

At last, for a small project (e.g. of less than a hundred thousand lines of source code), it is not very important. During the project you'll be quite easily able to refactor the code and reorganize it in different files. You'll just copy & paste chunks of code into new (header) files, not a big deal (what is more difficult is to design wisely how you would reorganize your files, and that is project specific).

^{(my personal preference is to avoid too large and too small files; I often have source files of several thousand lines each; and I am not afraid of a header file -including inlined function definitions- of a many hundred lines or even a couple of thousands of them)}

Notice that if you want to use pre-compiled headers with GCC (which sometimes is a sensible approach to lower compilation time) you need a single header file (including all the other ones, and the system headers also).

Notice that in C++, standard header files are pulling a lot of code. For example #include <vector> is pulling more than ten thousand lines on my GCC 6 on Linux (18100 lines). And #include <map> expands to almost 40KLOC. Hence, if you have many small header files including standard headers, you end up re-parsing many thousand of lines during build, and your compilation time suffers. This is why I dislike have many small C++ source lines (of a few hundred lines at most), but favor having fewer but larger C++ files (of several thousand lines).

^{(so having hundreds of small C++ files which always include -even indirectly- several standard header files gives huge build time, which annoys developers)}

In C code, quite often headers files expand to something smaller, so the trade-off is different.

Look also, for inspiration, into prior practice in existing free software projects (e.g. on github).

Notice that dependencies could be dealt with a good build automation system. Study the documentation of GNU make. Be aware of various -M preprocessor flags to GCC (useful to generate automatically dependencies).

In other words, your project (with less than a hundred files and a dozen of developers) is probably not big enough to be really concerned by "header hell", so your concern is not justified. You could have only a dozen of header files (or even much less), you could choose to have one header file per translation unit, you could even choose to have one single header file, and whatever you choose to do won't be a "header hell" (and refactoring and reorganizing your files would stay reasonably easy, so the initial choice is not really important).

^{(Don't focus your efforts on "header hell" -which is a non-issue for you-, but do focus them to design a good architecture)}