8

I'm learning how to build parsers using grammars, but I got stuck trying to express comments, because they can appear almost anywhere.

This indicates that comments can be stripped from the token stream before parsing takes place.

Is that the standard practice, or are comments ever/often specified in grammars?

4
  • "They can appear alomost everywhere" is somewhat of a stretch. There are quite a few languages that only allow comments between statements.
    – MSalters
    Jun 22, 2012 at 14:45
  • 1
    As an example of a language that does NOT remove the comments before parsing, see TCL - the comments are part of the grammar and if you put them in the wrong place, you can expect the parser to yell at you. Jun 22, 2012 at 18:32
  • @Michael I can't find a grammar for TCL, though: this page seems to indicate there isn't a static one.
    – user39685
    Jun 25, 2012 at 15:04
  • 1
    @MattFenwick - good gravy. I didn't know it was that malleable. According to what that says, you can't have a TCL grammer, it doesn't make sense. I had no idea, sorry for the false lead. Jun 25, 2012 at 17:27

4 Answers 4

2

It is very common to treat it as some form of white space. Much in the same way as newlines in semicolon-oriented languages like C.

Once it is some form of white-space, you frequently just ignore it higher in the parser.

17

They usually aren't.

They are removed by the lexer, when source code is transformed from characters to tokens.

Then, the parser will get tokens and build an AST. When the parser does its job, comment are already gone, so they don't have to appear in the grammar.

5

Quick Short Answer

Yes, its the standard practice, to detect comments in the "lexer" or the "parser"

Sometimes, the "parser" has a built-in "lexer" or is mixed with the "parser" as a single tool ("lexer-parser").

Extended Answer

I just working in that case.

Most "scanners" ( also know as "tokenizers" or "lexers" ), detect comments, but, removed when returning tokens to the "parser".

Sometimes, a programming language, uses some comments with a special meaning, like "compiler directives" or "documentation".

Standard Comment Example:

/*
 This function does something cool.
*/
int doSomething()
{
  return 0;
}

Directive Comment Example:

/*
 ##override
*/
int doSomething()
{
  return 0;
}

Comment for documentation generator Example:

/*
 @description: This Function text will be turn,
  into an external pdf file, togheter with other
  similar comments.
*/
int doSomething()
{
  return 0;
}

Most compiler-related tools, detect this special comments, with an analyzer or preprocessor, that is not the main lezxer or parser, and, even has a small lexer of its own.

Cheers.

0

Some parsers that retain comments and/or illegal inputs in the AST add them as properties to all nodes of the AST.

You could take a look at the Microsoft roslyn project which does that if I recall correctly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.