For sure, there is a well-known topic here on SE, however, it is almost 10 years old, and answers that were excellent by then, might be obsolete now and there could be modern easier ways (but I don't claim of that, just ask) .

Does anyone know, if during last 10 years, there was any significant advance, so we could have better & simpler ways to achieve that goal (to create a basic compiler for any basic or pseudo language), i.e. there more modern alternatives to lex/yacc, or easier tools.


The topic is being downvoted, because it's underrated, getting comments like: the principles are same or this question asks for opinions and such two words. They are not correct comments. The idea of this question is pretty simply can be reworded such - are there any newer programs than those referenced (lex & yacc ...) 10 years ago? So, you are saying me we have to use tools in nowadays programming, that was used 15 years ago and that we should follow the book about programming, which was written 25 years ago? As if you asked - in our school we used chalk on rough boards, is there any better approach nowadays? Is that opinion question? You have to answer For sure, there are newer plastic boards with temporary markers. This question is same.

  • 8
    No, the compiler fundamentals are the same as 10 years ago. – JacquesB Dec 16 '20 at 12:06
  • 1
    On the other hand, what we expect compilers to do, how we interact with them, and how we build them, has changed fundamentally in only the last few years. – Jörg W Mittag Dec 16 '20 at 12:47
  • 1
    I think there is a contratiction here. What differentiates the currently-modern compilers from those 15 or 20 years ago is precisely their complexity (e.g. complex type inference, layers upon layers of optimization, compiling the same language to vastly different targets, blazing-fast IDE interface, ...) – marstato Dec 16 '20 at 13:22
  • 6
    in other words, what marstato is saying is that basic compilers haven't changed. Complicated compilers have changed but you don't want to write a complicated compiler so it doesn't matter. – user253751 Dec 16 '20 at 14:32
  • 3
    No, that's not what they said. What they said is that, if you want to build a simple compiler, the principles from 10 years ago are still sound. If you want to build a modern compiler, it will no longer be simple. – Robert Harvey Dec 16 '20 at 16:17

Writing compilers in general is a solved problem. We have figured out the basic principles of how to convert a character stream into tokens into abstract syntax trees into assembly code into machine code, and design languages which are easy to create compilers for.

The Dragon Book (https://en.wikipedia.org/wiki/Compilers:_Principles,_Techniques,_and_Tools) from 1986 documents this very well, and is highly recommended.

I once read that compilers tend to be the most simple when there is zero, one or infinitely many of a given resource, like registers or stack space. This is also why modern compilers are rather complex (at least at the machine code generation stage) - namely because there are more than one but very less than infinitely many registers or cores (in a GPU) and generating the optimal code is simply hard. Especially because processors with the same instruction set may have different characteristics. The answers to https://stackoverflow.com/q/11227809/53897 not only demonstrate that different cpu's have different performance simply because of their branch prediction implementations, but also that different compilers can provide very different code - the Intel C++ compiler swapped two loops and avoided the problem asked about.

That said, what you are asking is if there has been any advances in tooling for creating compilers.

What you might be missing is that one of the most desired features for a compiler is that it is written in itself (because when you are there your toolchain becomes much simpler), and to my knowledge this is where most new languages (for which you need compilers) want to go, and usually do. So by implication modern tooling are written in that particular language, and therefore again usually not usable for other languages.

I would suggest you look into the new Roslyn compiler for C# because it is probably the newest really large scale open source compiler right now. Then you can see what their toolchain is today. (I have not looked at it).

So, what then if you just want to throw together a small, simple compiler. The traditional way is to use the Unix toolchain as reimplemented by the GNU project and widely available in Linux distributions, but you explicitly said you didn't want that.

Therefore I would suggest that you consider a modern Lisp dialect. This allows you to specify programs as Lisp-structures which can be directly read into memory (which is the same reason that JSON originally became popular) allowing you to start with the tokenized program instead of having to write a parser, which is a nice head start. Also Lisp programming might be quite a learning experience, as mentioned by Eric Raymond in http://www.catb.org/~esr/faqs/hacker-howto.html#skills1 . I looked at Scheme a while back and liked it, but I would suggest experimenting a bit to find the one you like the most. Look for a good debugger.

  • Thank you, nice answer! Btw, I would suggest something useful : post that answer into the old topic too: softwareengineering.stackexchange.com/questions/165543/… as my question was closed now, and your up-to-date answer will be a good gain in that topic – T.Todua Dec 17 '20 at 11:16
  • As a bonus, will be excellent, if you divide it into parts: for example, I concluded that I am interested only in "parsing" the custom language and tokenizing it. So, no machine-compiling needed for me. So, the first part mentioned which one (lex or yacc or whatever) are parsing tools, and the second part mentioned about compiling (which I am not aiming) – T.Todua Dec 17 '20 at 11:25
  • Why ask for tools to write a compiler if you don’t want to compile? – Thorbjørn Ravn Andersen Dec 17 '20 at 12:38
  • You really should consider writing better questions. You’ve basically wasted my time. – Thorbjørn Ravn Andersen Dec 18 '20 at 10:41
  • No Thor, you perfectly helped. After you posted answer, i've decided which compiler I had to choose. And moreoever, as i suggested , if you posted answer in the second topic too, would be useful for many people. – T.Todua Dec 18 '20 at 10:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.