Your grammar has ambiguities that make it impossible to know what to do with, say, the letter a
without context. In your case, the string abc
can have two interpretations: it can be an identifier (I'm assuming that's what your first m//
defines), or it can be part of a string literal quoted in your { ... }
notation (I'll call that a "quoted list"). Lexical analyzers (tokenizers) aren't smart enough to handle that kind of ambiguity, because their concept of context is very simplistic. Parsers, on the other hand, can understand context at very deep levels.*
Language designers sometimes add sigils to their identifiers (e.g., $abc
) to make them easier to tokenize. This is why you can have a Perl variable named $for
even though bare-naked for
has special meaning. For similar reasons, C lexers tokenize /"[^"]*"/
into a string literal because it has a context-independent syntax that doesn't appear anywhere else in the language.
Back to your problem: Prematurely tokenizing a string of alphanumerics into an IDENTIFIER
would mean the quoted list { abc[1]xyz }
would be fed to the parser as {
IDENTIFIER
[
NUMBER
]
IDENTIFIER
}
. That's useful if those were the chunks in which you needed it, but you'd otherwise have to incorporate being able to handle combining all combinations of those tokens into the grammar for your quoted list. Then you'd have to handle reassembling them back into string literals. If you haven't guessed by now, that would get complex and ugly very quickly. But because parsers understand context, putting that wisdom there makes it clean and easy.
For what you're doing, there shouldn't be much of a tokenizer at all because so much of it is context-sensitive, and that's all parser territory. Whitespace doesn't seem to matter except in the context of a quoted list, so you could tokenize that as well as things that aren't ambiguous like LETTER
and DIGIT
.
// NOTE: This code doesn't handle the case where whitespace is
// interspersed with the tokens. See the comments.
quoted-list ::= '{' quoted-list-item-set '}'
quoted-list-item-set ::=
<nothing>
| string-of-non-whitespace-characters
| string-of-non-whitespace-characters WHITESPACE quoted-list-item-set
// This ends up being things you have to put together and return,
// so that eventually you end up with a single string.
string-of-non-whitespace-characters ::=
non-whitespace-character
| non-whitespace-character string-of-non-whitespace-characters
non-whitespace-character ::= <anything in the set '!'..'~'>
identifier ::= LETTER alphanumeric-string
alphanumeric-string ::=
<nothing>
| alphanumeric alphanumeric-string
alphanumeric ::= LETTER | DIGIT
// ...etc...
// This prevents the parser from barfing on whitespace in any other context.
things-that-get-ignored ::= WHITESPACE
*This is why you should use a parser to interpret something complex like XML and not fall into the trap of trying to understand it with regular expressions.
IF MAN IS TALL THEN SHORTEN HIM
is a problem cause it can't tell ifIS
is part of theIF
statement or not. Same for open braces.IF
and theTHEN
as the conditional? And I'm not sure what you mean about your history idea.