This started out as a SO question but I realized that it is quite unconventional and based on the actual description on the websites, it might be better suited to programmers.se since the question has a lot of conceptual weight.
I have been learning clang LibTooling and it is a very powerful tool capable of exposing the entire "nitty gritty" of the code in a friendly way, that is, in a semantic way, and not by guessing either. If clang can compile your code, then clang is certain about the semantics of every single character inside that code.
Now allow me to step back for a moment.
There are many practical problems that arise when one engages in C++ template metaprogramming (and especially when venturing beyond templates into the territory of clever albeit terrifying macros). To be honest, to many programmers, myself included, many of the ordinary uses of templates are also somewhat terrifying.
I guess a good example would be compile-time strings. This is a question that is over a year old now, but it is clear that C++ as of right now does not make this easy for mere mortals. While looking at these options isn't quite enough to induce nausea for me, it nevertheless leaves me unconfident about being able to produce magical, maximally efficient machine code to suit whatever fancy application I have for my software.
I mean, let's face it, folks, strings are pretty simple and basic. Some of us just want a convenient way to emit machine code that has certain strings "baked in" significantly more than we do get when coding it the straightforward way. In our C++ code.
Enter clang and LibTooling, which exposes the abstract syntax tree (AST) of the source code and allows a simple custom C++ application to correctly and reliably manipulate raw source code (using Rewriter) alongside a rich semantic object-oriented model of everything in the AST. It handles a lot of things. It knows about the macro expansions, and lets you follow those chains. Yes, I am talking about source-to-source code transformation or translation.
My fundamental thesis here is that clang now enables us to create executables which themselves can function as the ideal custom preprocessor stages to our C++ software, and we can implement these metaprogramming stages with C++. We are simply constrained by the fact that this stage must take input which is valid C++ code and produce as output more valid C++ code. Plus whatever other constraints your build system applies.
The input has to be at least very close to valid C++ code because, after all, clang is the compiler front-end and we are just poking around and being creative with its API. I do not know if there is any provision for being able to define new syntax to use, but clearly we have to develop the ways to properly parse it and add it to the clang project in order to do this. To expect any more is to have something in the clang project that is out of scope.
Not a problem. I would imagine that some no-op macro functions can handle this task.
Another way to look at what I'm describing is to implement metaprogramming constructs using runtime C++ by manipulating the AST of our source code (thanks to clang and its API) instead of implementing them using the more limited tools available in the language itself. This has clear compilation performance benefits as well (template-heavy headers slow compilation proportionally to how often you use them. Lots of compiled stuff then gets carefully matched up and thrown away by the linker).
This does, however, come at the cost of introducing an additional step or two in the build process and also in the requirement of writing some (admittedly) somewhat more verbose software (but at least it is straightforward runtime C++) as part of our tool.
That isn't the whole picture. I am pretty certain that there is a much larger space of functionality that can be had from generating code that is extremely difficult or impossible with core language features. In C++ you can write a template or a macro or a crazy combination of both, but in a clang tool you can modify classes and functions in ANY way that you can achieve with C++, at runtime, while having full access to the semantic content, in addition to template and macros and everything else.
So, I'm wondering about why everybody isn't already doing this. Is it that this functionality from clang is so new and nobody is familiar with the huge class hierarchy of clang's AST? That can't be it.
Perhaps I am just underestimating the difficulty of this a little bit, but doing "compile-time string manipulation" with a clang tool is nearly criminally simple. It's verbose, but it's insanely straightforward. All that's needed are a bunch of no-op macro functions that map to actual real std::string operations. The clang plugin implements this by fetching all the relevant no-op macro calls, and performs the operations with strings. This tool is then inserted as a part of the build process. During build, these no-op macro function calls are automatically evaluated into their results, and then inserted back as plain old compile-time strings in the program. The program can then be compiled as usual. In fact this resulting program is also much more portable as a result, not requiring a fancy new compiler supporting C++11.
