Should I use a source-to-source or a traditional compiler in order to develop my own Programming Language?

Question

I'm really interested in writing my own general-purpose high-level programming language, but I'm somewhat confused.

I know that Python and Ruby were written in C, which makes me wonder that if I want to write my 'own Python', is it preferable to use a source-to-source compiler in order to translate all the source code of my language to C, or should I target assembly language?

The point is, I know that I need to dig in Compiler Design and understand Lexical Analysis, and the whole process of parsing the code and generating tokens and an intermediate language and checking for syntax and semantic errors and then generating output code.

However, since I'm not an expert in low level / assembly programming, should I use a source-to-source compiler? What challenges might I face if I attempt to compile my language to assembly? What drawbacks might there be in using a source-to-source compiler? What domain specific facets of my situation should I consider when making this decision?

Answer 1

I am going to focus on your core question, since other stuff has been answered elsewhere.

Should you target a higher level language or assembly?

Getting software done is hard. While making a new language can be pretty easy, you need to stick to simple stuff and avoid the stuff that is a pain to implement. Making your first language has the problem that you don't know what the "pain to implement" stuff is. And let's face it, you're not aiming making a new language that can only implement console based calculators - the interesting stuff is going to be non-trivial to implement.

So do yourself a favor and target a language that you already know. Making a new language and a functional compiler is hard enough without adding "learn assembly" to the list of tasks. By setting yourself up for success, you're more likely to have fun and learn from the endeavor.

Answer 2

You should consider compiling to C (see this and that answers), or to some other languages (Java, Common Lisp, Ocaml, C++, or even a mix of Javascript & C - like HOP does... Notice that RefPerSys is generating C++ code, or in 2022 C code here). You could also compile to the textual representation of LLVM bytecode, or use the LLVM library as your backend, or (if having GCC 5 or better) use the libgccjit (to target GCC internal representations and profit from GCC optimizations). You could also choose some existing bytecode (e.g. JVM, Ocaml, Neko, Parrot, ...) and compile to it. And you might also use some JIT library like libjit, GNU lightning, asmjit etc...

Lexing & parsing are not the main work of a compiler or an interpreter. They are the simple parts. A compiler is mostly transforming (very often in several passes) some internal representations (in particular Abstract Syntax Trees, but not only them) of the source code that it is compiling. An interpreter is often transforming some internal representations, then traversing others (e.g. some bytecode, or some normalized AST). Play with GCC's -fdump-tree-all option, and perhaps with MELT (a Lisp-like DSL to inspect and/or transform GCC internal representations). The semantics of your programming language matters more than syntax.

An important part is memory management. Do you want a garbage collector (it is a core part of your semantics)? What about typing (static or dynamic) & type inference? Do you handle tail-calls? Do you want homoiconicity? metaprogramming? Do you want closures (they most often need a GC)? Consider Boehm's conservative GC, and/or read the GC handbook.

Bootstrapping compilers is important. See also this and the references I gave there. Read also this & that answers explaining technical & practical details (and should heal your headache about "Haskell written in Haskell", "Ocaml written in Ocaml", "MELT written in MELT", "CAIA written in CAIA", "GCC or Clang/LLVM written in C++").

Also, if you know none of them, play with Ocaml, or Common Lisp, or Haskell, or Scheme (see also SICP). Read Scott's book on Programming Language Pragmatics and Queinnec's book on Lisp In Small Pieces and Pitrat's book Artificial Beings, the conscience of a conscious machine (and blog here).

Be sure to make your language implementation some free software (on http://github.com/ you'll find many other language implementations - e.g. compilers or interpreters).

Answer 3

I did some professional work three years ago that used reflection to understand a .NET interface and provide some CIL glue to mangle it onto a base class. It was an eye opener to the level of additional work required.

Most software development, you focus on the success routes and if something unexpected happen you catch an exception. I found that the success route was less than 20% of the work. The usual strategy of catching exceptions doesn't work, as the exceptions occur at run time, not when you are generating the code. Instead you have to think of and check for any possible combination that could break your code, and then either support it, or fail the compilation. This is very different from application development.

If you are wanting to create a new programming language, and want it to be adopted by others, your best bet is to find a problem that cannot currently be easily solved. If you create a new programming language that provides a clean and simple approach to solve a complex problem, then people with that problem have a good reason to adopt your language.

Answer 4

"I'm really interested in writing my own general-purpose high-level programming language" Are you really? If you are truly interested and want to create your own language complete with a (native) compiler for the experience you would probably be interested on reading up on assembly and how the CPU works, you can examine the assembly output of existing high level languages to give you an idea.

A source to native machine code compiler gives you a lot more freedom to define your language's semantics, if you compile to C you are basically limited to the C way of doing things, so in the end features you may want to include such as proper tail calls will be out of your reach. Building a native compiler is also likely to be a more rewarding and enlightening experience, whereas compiling to C you are basically using something somebody else built C is a very complicated language for something to output to (it's syntax is designed to be good for humans to write not easy for machines to generate), you have to worry about nesting order of declarations, making sure you specify enough information to allow the C compiler to optimize your code effectively (e.g. using restrict when there is no aliasing of pointers).

If you're pragmatic and want to build a language which is going to be used in the near future and you're not really interested in building the compiler as much having a working compiler for your language then source to source is the way to go, however be aware that your language effectively becomes a preprocessor (in-fact a preprocessor is good place to start learning about lexical analysis and parsing). Either way if your goal is to distribute your language try to avoid it becoming a JAP language (Just another programming language) that is a language which offers nothing new and isn't a major improvement over existing languages.

Answer 5

If you translate whatever you have to C, then you have a working system wherever a C compiler is available. Which is basically everywhere. You don't have to worry about different processors, different operating systems, and all the rest.

You may instead compile to C++, which today is almost as common as C, and gives you the advantage that you can hide work that you need to do in a class instead of having to repeat yourself. Important if your language has objects that cannot be translated to C or C++ primitives.

Answer 6

If you target LLVM, you still need to preprocess and actually compile the input, but output can then be in any of the outputs LLVM allows, be it assembly or JavaScript.