I have recently caught the 'Toy Language' bug, and have been experimenting with various simple tokenizer configurations. The most recent one, makes use of the boost.regex library to identify and get the value of tokens. It seems to me that regex would be the preferred way to go, when creating a tokenizer. My research has proved my assumption to be false however, or at least not completely true.
This is what a user from Stack Overflow had to say regarding a question about: Is it bad idea using regex to tokenize string for lexer?:
Using regular expressions is THE traditional way to generate your tokens.
After doing more research into this topic, I realized that even the most popular lexer generators use regex. For example, take the lexer generator Lex:
Lex helps write programs whose control flow is directed by instances of regular expressions in the input stream. It is well suited for editor-script type transformations and for segmenting input in preparation for a parsing routine. - http://dinosaur.compilertools.net/lex/.
That lead me to draw the conclusion that regex is the usual, preferred way to create a tokenizer. After researching my topic once more, and trying to find a second opinion however, I came across this statement in response to this question on Stack Overflow:
The reason why people tell you that regular expressions aren't the best idea for a case like this is because regular expressions take more time to evaluate, and the regular expression language has many limitations and quirks that make it unsuitable for many applications. - user:670358
And he went on to say
Many compilers use a basic single-pass tokenization algorithm. The tokenizer has a very basic idea of what can be used as a delimiter, how constants and identifiers should be treated, etc. The tokenizer will then just iterate through the input quickly and emit a string of tokens that can then be easily parsed. - user:670358
This left me both confused and contemplating several things:
- Is regex necessary for a hand built tokenizer. Or is it overkill?
- If regex is not used often, then how extacly are the tokens parsed?
This may or may not opinion based, but I believe that there is a preferred method/accepted method among programmers.