Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
I'm learning how to build parsers using grammars, but I got stuck trying to express comments, because they can appear almost anywhere.
This indicates that comments can be stripped from the token stream before parsing takes place.
Is that the standard practice, or are comments ever/often specified in grammars?
It is very common to treat it as some form of white space. Much in the same way as newlines in semicolon-oriented languages like C.
Once it is some form of white-space, you frequently just ignore it higher in the parser.
They usually aren't.
They are removed by the lexer, when source code is transformed from characters to tokens.
Then, the parser will get tokens and build an AST. When the parser does its job, comment are already gone, so they don't have to appear in the grammar.
Quick Short Answer
Yes, its the standard practice, to detect comments in the "lexer" or the "parser"
Sometimes, the "parser" has a built-in "lexer" or is mixed with the "parser" as a single tool ("lexer-parser").
Extended Answer
I just working in that case.
Most "scanners" ( also know as "tokenizers" or "lexers" ), detect comments, but, removed when returning tokens to the "parser".
Sometimes, a programming language, uses some comments with a special meaning, like "compiler directives" or "documentation".
Standard Comment Example:
/*
This function does something cool.
*/
int doSomething()
{
return 0;
}
Directive Comment Example:
/*
##override
*/
int doSomething()
{
return 0;
}
Comment for documentation generator Example:
/*
@description: This Function text will be turn,
into an external pdf file, togheter with other
similar comments.
*/
int doSomething()
{
return 0;
}
Most compiler-related tools, detect this special comments, with an analyzer or preprocessor, that is not the main lezxer or parser, and, even has a small lexer of its own.
Cheers.
Some parsers that retain comments and/or illegal inputs in the AST add them as properties to all nodes of the AST.
You could take a look at the Microsoft roslyn project which does that if I recall correctly.
