This extends off this other Q&A thread, but is going into details that are out of scope from the original question.

I am generating a parser that is to parse a context-sensitive grammar which can take in the following subset of symbols: ,, [, ], {, }, m/[a-zA-Z_][a-zA-Z_0-9]*/, m/[0-9]+/

The grammar can take in the following string { abc[1] }, } and parse it as ({, abc[1], }, }). Another example would be to take: { abc[1] [, } and parse it as ({, abc[1], [,, }).

This is similar to the grammar used in Perl for the qw() syntax. The braces indicate that the contents are to be whitespace tokenized. A closing brace must be on its own to indicate the end of the whitespace tokenized group. Can this be done using a single lexer/tokenizer, or would it be necessary to have a separate tokenizer when parsing this group?

  • I don't think lexer can repair syntax when braces are missing. The tokenizer would be left in an invalid state, and token matching won't be performed.
    – Reactgular
    May 16, 2013 at 16:43
  • @Mathew, don't you mean that the parser would be left in an invalid state? The lexer is just a token generator. If so, I agree.
    – Adrian
    May 16, 2013 at 17:28
  • It's been awhile since I last played with this kind of parser, but they need strict structure to work. Languages where parameters are ambiguous won't work. Like IF MAN IS TALL THEN SHORTEN HIM is a problem cause it can't tell if IS is part of the IF statement or not. Same for open braces.
    – Reactgular
    May 16, 2013 at 17:36
  • You would need a streaming parser that is history dependant to make it work, but those are difficult to get working because they can be confused easily.
    – Reactgular
    May 16, 2013 at 17:38
  • @MathewFoscarini, wouldn't that example you just stated be easily parse-able(?) if you made your grammar rules to take whatever is between the IF and the THEN as the conditional? And I'm not sure what you mean about your history idea.
    – Adrian
    May 16, 2013 at 17:50

4 Answers 4


Your grammar has ambiguities that make it impossible to know what to do with, say, the letter a without context. In your case, the string abc can have two interpretations: it can be an identifier (I'm assuming that's what your first m// defines), or it can be part of a string literal quoted in your { ... } notation (I'll call that a "quoted list"). Lexical analyzers (tokenizers) aren't smart enough to handle that kind of ambiguity, because their concept of context is very simplistic. Parsers, on the other hand, can understand context at very deep levels.*

Language designers sometimes add sigils to their identifiers (e.g., $abc) to make them easier to tokenize. This is why you can have a Perl variable named $for even though bare-naked for has special meaning. For similar reasons, C lexers tokenize /"[^"]*"/ into a string literal because it has a context-independent syntax that doesn't appear anywhere else in the language.

Back to your problem: Prematurely tokenizing a string of alphanumerics into an IDENTIFIER would mean the quoted list { abc[1]xyz } would be fed to the parser as { IDENTIFIER [ NUMBER ] IDENTIFIER }. That's useful if those were the chunks in which you needed it, but you'd otherwise have to incorporate being able to handle combining all combinations of those tokens into the grammar for your quoted list. Then you'd have to handle reassembling them back into string literals. If you haven't guessed by now, that would get complex and ugly very quickly. But because parsers understand context, putting that wisdom there makes it clean and easy.

For what you're doing, there shouldn't be much of a tokenizer at all because so much of it is context-sensitive, and that's all parser territory. Whitespace doesn't seem to matter except in the context of a quoted list, so you could tokenize that as well as things that aren't ambiguous like LETTER and DIGIT.

// NOTE:  This code doesn't handle the case where whitespace is 
// interspersed with the tokens.  See the comments.

quoted-list ::= '{' quoted-list-item-set '}'

quoted-list-item-set ::=
    | string-of-non-whitespace-characters
    | string-of-non-whitespace-characters WHITESPACE quoted-list-item-set

// This ends up being things you have to put together and return,
// so that eventually you end up with a single string.
string-of-non-whitespace-characters ::=
    | non-whitespace-character string-of-non-whitespace-characters

non-whitespace-character ::= <anything in the set '!'..'~'>

identifier ::= LETTER alphanumeric-string

alphanumeric-string ::= 
    | alphanumeric alphanumeric-string

alphanumeric ::= LETTER | DIGIT

// ...etc...

// This prevents the parser from barfing on whitespace in any other context.
things-that-get-ignored ::= WHITESPACE

*This is why you should use a parser to interpret something complex like XML and not fall into the trap of trying to understand it with regular expressions.

  • What grammar parser are you using? AFAIK, yacc and bison are not able to do this. Also, abc in the context of a quoted list can only be a string. Identifiers can only exist outside of the quoted list. One other thing, what I stated is a small subsection of the grammar that is in use, which I think you are addressing in your last line of 'code', but I'm not sure how.
    – Adrian
    May 16, 2013 at 18:22
  • The notation above is just rough BNF. I haven't turned any of this into code, but YACC or Bison should be able to handle this easily. I'll add a clarification to my answer to cover a couple of things I overlooked.
    – Blrfl
    May 16, 2013 at 18:58
  • Also for completeness, how are you coding the string-of-non-whitespace-characters grammar rule?
    – Adrian
    May 16, 2013 at 19:02
  • @Adrian: Hopefully what I added will help.
    – Blrfl
    May 16, 2013 at 19:50
  • Ok. I'm unsure of one thing. Can a literal stated in the grammar override the tokenizer's rules? If for instance, the tokenizer recognizes '!', will your string-of-non-whitespace-characters work? If it does work, specifying all the different characters could be problematic, or at least inconvenient, which could be super inconvenient if using a wide character set. I don't think there is a negation rule like not whitespace.
    – Adrian
    May 16, 2013 at 20:00

Yes, it's certainly possible to create a single tokenizer which can parse that. I can easily create a context-free regular tokenizer which will correctly parse your language.

However, many popular tools may make it hard and/or impossible to tokenize your input the way you want, even if your tokenizer could be described by a regular grammar. Using different tools, you may find it extremely easy to tokenize your input any way you want.

What approach you take to solve this problem may be largely dictated to you by your choice of tool.

  • What is a regular tokenizer? I've never heard of one before, nor can I find any reference to such a beast on the web.
    – Adrian
    May 16, 2013 at 18:52
  • 1
    It's just a tokenizer that works on a regular language, as opposed to a context-free, context-sensitive, or unrestricted language. Are you familiar with the different types of languages?
    – user39685
    May 16, 2013 at 19:10
  • I am, but it's been a while since I've done so formally in university. Still, even in that link, I wasn't able to find reference to a tokenizer that is specific to a grammar or language. Though there was reference to a regular language being able to be obtained by a regular-expression. But I'm not sure as to what that means exactly.
    – Adrian
    May 16, 2013 at 19:27

From what I see all it takes is making lexer to recognize opening brace and switching into greedy state. In such state you define new set of patterns, which are, well, greedy except for whitespaces. And single right brace has the effect of popping the state and going back to what was before.

The described approach would correctly tokenize both your examples.


As I understand it, lexers are non-discriminatory. They break up tokens without context. Thus the first example cannot be parsed using that tokenizer as the closing brace is not tightly bound to the comma. The second example cannot be parsed that way, even if you tried to paste tokens together from within the parser (adding overhead and complexity), because there would not be a way to know when the first token ends and the next begins.

Both could be achieved if whitespaces were not ignored, but would add more complexity to the grammar/parser for all the other grammar rules.

So yes, it can be done. But it depends on if the added overhead and complexity is worth it.

  • 2
    You seem to be asking these questions with a certain technology in mind, although I haven't seen you mention what exactly that technology is. As I tried to point out in my answer, the technology you use plays a big role in how you're allowed to solve the problem; thus, it's important to take a step back and understand the problem without worrying about artificial constraints of specific technologies.
    – user39685
    May 16, 2013 at 18:49
  • The only technologies I have in mind are those that are commonly used, such as yacc, bison, or hand crafting a recursive decent parser. The question is based on if, when writing a parser to parse said code fragment, would a separate lexer be required for that fragment.
    – Adrian
    May 16, 2013 at 19:00
  • You should do more research; there are far more technologies for creating parsers.
    – user39685
    May 16, 2013 at 19:09
  • That's possible. I have looked around for parsers besides yacc and bison, but I was unable to find a search query that would give me anything other than command line parsers and other things that were irrelevant. Perhaps you could point me in the right direction?
    – Adrian
    May 16, 2013 at 19:17
  • 2
    Parser combinators are a huge one -- tons of libraries in every language for 'em. Antlr is quite popular and has a good reputation; haven't used it much so not sure how different it is from lex/yacc though.
    – user39685
    May 16, 2013 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.