When does it make sense to compile my own language to C code first?

Question

When designing an own programming language, when does it make sense to write a converter that takes the source code and converts it to C or C++ code so that I can use an existing compiler like gcc to end up with machine code? Are there projects that use this approach?

Answer 1

Tranlating to C code is a very well established habit. The original C with classes (and the early C++ implementations, then called Cfront) did that successfully. Several implementations of Lisp or Scheme are doing that, e.g. Chicken Scheme, Scheme48, Bigloo. Some people translated Prolog to C. And so did some versions of Mozart (and there have been attempts to compile Ocaml bytecode to C). J.Pitrat's artificial intelligence CAIA system is also bootstrapped and generates all its C code. Vala also translates to C, for GTK related code. Queinnec's book Lisp In Small Pieces have some chapter about translation to C.

One of the issues when translating to C is tail-recursive calls. The C standard does not guarantee that a C compiler is translating them properly (to a "jump with arguments", i.e. without eating call stack), even if in some cases, recent versions of GCC (or of Clang/LLVM) do that optimization.

Another issue is garbage collection. Several implementations just use the Boehm conservative garbage collector (which is C friendly...). If you wanted to garbage collect code (like several Lisp implementations do, e.g. SBCL) that might be a nightmare (you would like to dlclose on Posix).

Yet another issue is dealing with first-class continuations and call/cc. But clever tricks are possible (look inside Chicken Scheme). Accessing the call-stack could require a lot of tricks (but see GNU backtrace, and libbacktrace etc....). Orthogonal persistence of continuations (i.e. of stacks or threads) would be difficult in C.

Exception handling is often a matter to emit clever calls to longjmp etc...

You may want to generate (in your emitted C code) appropriate #line directives. This is boring and takes a lot of work (you'll want that to e.g. produce more easily gdb-debuggable code).

My obsolete GCC MELT lispy domain specific language (to customize or extend GCC) is translated to C (actually to poor C++ now). It has its own generational copying garbage collector. (You might be interested by Qish or Ravenbrook MPS). Actually, generational GC is easier in machine generated C code than in hand-written C code (because you'll tailor your C code generator for your write-barrier and GC machinery). The Bismon static source code analyzer (described in this DRAFT report) also generates C code.

I don't know any language implementation translating to genuine C++ code, i.e. using some "compile-time garbage collection" technique to emit C++ code using a lot of STL templates and respecting the RAII idiom. (please tell if you know one). But RefPerSys aims to generate C++ code (or C code, perhaps also machine code).

What is funny today is that (on current Linux desktops) C compilers may be fast enough to implement an interactive top level read-eval-print-loop translated to C: you'll emit C code (a few hundred lines) at every user interaction, you'll fork a compilation of it into a shared object, which you would then dlopen. (MELT is doing that all ready, and it is usually fast enough). All this might take a few tenths of a second and be acceptable by end-users.

When possible, I would recommend translating to C, not to C++, in particular because C++ compilation is slow. However, C++ has today powerful standard containers, exceptions, λ-expressions, etc .... and is used or required by interesting C++ libraries or frameworks such as Qt, POCO, Tensorflow, and all these features is what motivates the choice of generating C++ code in a pet project of mine called RefPerSys. If generating C++ dynamically, either accept to wait more than a second for compiling every generated C++ file (e.g. into a temporary plugin, see for Linux the C++ dlopen mini howto) or use clever tricks (e.g. ccache and/or GCC pre-compiled headers, etc....) while minimizing if possible the total amount of #include-d material) to decrease the C++ compilation time.

If you are implementing your language, you might also consider (instead of emitting C code) some JIT libraries like libjit, GNU lightning, asmjit, or even LLVM or GCCJIT. If you want to translate to C, you might sometimes use tinycc: it compiles very quickly the generated C code (even in memory) to slow machine code. But in general you want to take advantage of the optimizations done by a real C compiler like GCC

If you translate to C your language, be sure to build the entire AST of the generated C code in memory first (this also makes easier to generate first all the declarations, then all the definitions and function code). You would be able to do some optimizations/normalizations this way. Also, you could be interested in several GCC extensions (e.g. computed gotos). You'll probably want to avoid generating huge C functions - e.g. of a hundred thousands line of generated C - (you'll better split them into smaller pieces) since optimizing C compilers are very unhappy with very big C functions (in practice, and experimentally, gcc -O compilation time of large functions is proportional to the square of the function code size). So limit the size of your generated C functions to a few thousand lines each.

Notice that both Clang (thru LLVM) and GCC (thru libgccjit) C & C++ compilers offer some way to emit some internal representations suited for these compilers, but doing so might (or not) be harder than emitting C (or C++) code, and is specific to each compiler. Also, recent GCC are extensible thru dlopen-ed plugins.

If designing a language to be translated to C, you probably want to have several tricks (or constructs) to generate a mixture of C with your language. My DSL2011 paper MELT: a Translated Domain Specific Language Embedded in the GCC Compiler should give you useful hints.

Generating C code might be done with GNU m4 (like GNU autoconf does). But you could also consider using GNU guile, Bigloo, SBCL, Ocaml, Python, GNU bison, GPP, etc... when coding your own C code generator.

Answer 2

It makes sense when the time to generate full machine code outweighs the inconvenience of having an intermediate step of compiling your "IL" into machine code using a C compiler.

Typically domain-specific languages are written in this way, a very high level system is used to define or describe a process that is then compiled into an executable or dll. The time taken to produce working/good assembly is much greater than generating C, and C is quite close the assembly code for performance, so it makes good sense to generate C and re-use the skills of the C compiler writers. Note that it isn't just compiling, but optimising too - the guys who write gcc or llvm have spent a lot of time making optimised machine code, it'd be daft to try to reinvent all their hard work.

It might be more acceptbale to re-use LLVM's compiler backend which IIRC is language neutral, so you generate LLVM instructions instead of C code.

Answer 3

Writing a compiler to produce machine code may not be much more difficult than writing one which produces C (in some cases it may be easier), but a compiler which produces machine code will only be able to produce runnable programs on the particular platform for which it was written; a compiler that produces C code, by contrast, may be able to produce program for any platform which uses a dialect of C which the generated code is designed to support. Note that in many cases it may be possible to write C code which is completely portable and which will behave as desired without using any behaviors not guaranteed by the C standard, but code which relies upon platform-guaranteed behaviors may be able to run much faster on platforms which make those guarantees than code which does not.

For example, suppose a language supports a feature to yield a UInt32 from four consecutive bytes of an arbitrarily-aligned UInt8[], interpreted in big-endian fashion. On some compilers, one could write the code as:

uint32_t dat = *(__packed uint32_t*)p;
return (dat >> 24) | (dat >> 8) | ((uint32_t)dat << 8) | ((uint32_t)dat << 24));

and have the compiler generate a word-load operation followed by a reverse-bytes-in-word instruction. Some compilers, however, would not support the __packed modifier and in its absence would generate code that wouldn't work.

Alternatively, one could write the code as:

return dat[3] | ((uint16_t)dat[2] << 8) | ((uint32_t)dat[1] << 16) | ((uint32_t)dat[0] << 24);

such a code should work on any platform, even those where CHAR_BITS isn't 8 (assuming that each octet of source data ended up in a distinct array element), but such code may likely not run nearly as fast as the would the non-portable version on platforms supporting the former.

Note that portability often requires that code be extremely liberal with typecasts and similar constructs. For example, code which wants to multiply two 32-bit unsigned integers and yield the lower 32 bits of the result must for portability be written as:

uint32_t result = 1u*x*y;

Without that 1u, a compiler on a system where INT_BITS ranged from 33 to 64 could legitimately do anything it wanted if the product of x and y was larger than 2,147,483,647, and some compilers are prone to take advantage of such opportunities.

Answer 4

You have some excellent answers above but given that, in a comment, you answered the question "Why do you want to create a programming language of your own in the first place? " with "It would be for learning purpose mainly," I'm going to answer from a different angle.

It makes sense to write a converter that takes the source code and converts it to C or C++ code, so that you can use an existing compiler like gcc to end up with machine code, if you are more interested in learning about lexical, syntax and semantic analysis than you are in learning about code generation and optimization!

Writing your own machine code generator is a pretty significant piece of work which you can avoid by compiling to C code, if it's not what you are primarily interested in!

If, however, you are into assembly program and fascinated by the challenges of optimizing code at the lowest level, then by all means, write a code generator yourself for the learning experience!

Answer 5

It depends on what Operating System you are using if you are using Windows there is a Microsoft IL (Intermediate Language) Which converts your code into intermediate language so that it takes no time to get compiled into machine code. Or If you are using Linux there is a separate compiler for that

Coming back to your question is when you when designing your own language you should have a separate compiler or interpreter for that because machine does not know the high level language . Your code should be compiled into machine code to make it useful for machine