I am currently developing my own programming language (written in Lua). The text analyzer is composed of two functions:
lexer takes as input a string, cycles over each character of it and divides it in tokens, either single words, numbers, strings, or symbols. It then returns an array of them.
parser takes a string too, internally calls
lexer over it, and then, with the help of some recursive helper functions, builds a tree structure over the array returned from
Now, this seems like a good example where the use of Lua's coroutines would fit very well:
parser, as a normal function, would feed characters to the
lexer coroutine, which waits until it has enough characters to return a token. Then
parser uses that token for its tree structure.
This would require a bit of extra work to convert my code, but the question is: is it worth it? The code would probably consume less memory, but what are the other advantages? Would it be faster? More readable (probably not)? Should I overall do it?