How are comments usually parsed?

Question

How are comments generally treated in programming languages and markup? I am writing a parser for some custom markup language and want to follow the principle of least surprise, so I'm trying to determine the general convention.

For example, should a comment embedded within a token 'interfere' with the token or not? Generally, is something like:

Sys/* comment */tem.out.println()

valid?

Also, if the language is sensitive to new lines, and the comment spans the new line, should the new line be considered or not?

stuff stuff /* this is comment
this is still comment */more stuff 

be treated as

stuff stuff more stuff

or

stuff stuff
more stuff

?

I know what a few specific languages do, nor am I looking for opinions, but am looking for whether or not: is there a general consensus what is generally expected by a mark up in regards to tokens and new lines?

---

My particular context is a wiki-like markup.

Answer 1

Usually comments are scanned (and discarded) as part of the tokenization process, but before parsing. A comment works like a token separator even in the absence of whitespace around it.

As you point out, the C specification explicitly states that comments are replaced by a single space. It is just specification-lingo though, since a real-world parser will not actually replace anything, but will just scan and discard a comment the same way it scans and discards whitespace characters. But it explains in a simple way that a comment separates tokens the same way a space would.

The content of comments are ignored, so linebreaks inside multiline comments have no effect. Languages which are sensitive to line breaks (Python and Visual Basic) usually do not have multiline comments, but JavaScript is one exception. For example:

return /*
       */ 17

Is equivalent to

return 17

not

return
17

Single-line comments preserve the line break, i.e.

return // single line comment
    17

is equivalent to

return
17

not

return 17

Since comments are scanned but not parsed, they tend not to nest. So

 /*  /* nested comment */ */

is a syntax error, since the comment is opened by the first /* and closed by the first */

Answer 2

To answer the question:

is there a general consensus what is generally expected by a mark up?

I would say none would expect a comment embedded inside of a token to be legal.

As a general rule of thumb, comments should be treated the same as whitespace. Any place that would be valid to have extraneous whitespace should also be allowed to have an embedded comment. The only exception would be strings:

trace("Hello /*world*/") // should print Hello /*world*/

It would be quite odd to support comments inside strings, and would make escaping them tedious!

Answer 3

In whitespace-insensitive languages, ignored characters (i.e. whitespaces or those that are part of a comment) delimit tokens.

So for example Sys tem are two tokens, while System is one. The usefulness of this might be more apparent if you compare new Foo() and newFoo() one of which will construct an instance of Foo while the other calls newFoo.

Comments can play the same role as a run of whitespaces, e.g. new/**/Foo() works the same as new Foo(). Of course this can be more complex, e.g. new /**/ /**/ Foo() or whatnot.

Technically, it should be possible to allow comments within identifiers, but I doubt it is particularly practical.

Now, what of white-space sensitive languages?

Python comes to mind and it has a very simple answer: no block comments. You start a comment with # and then the parser works exactly as though the rest of the line didn't exist but were just a newline instead.

In contrast to that, jade allows for block comments, where the block ends when you get back to the same indentation level. Example:

body
  //-
    As much text as you want
    can go here.
  p this is no longer part of the comment

So in this realm, I wouldn't say you could say how things are usually handled. What seems to be a commonality, is that a comment always ends with an end-of-line, which means that all comments act exactly the same as new lines.

Answer 4

In the past I've turned comments into a single token as a part of the lexical analysis. The same goes for strings. From there, life is easy.

In the specific case of the last parser I built, an escape rule is passed to the top level parse routine. The escape rule is used to handle tokens such as comment tokens inline with the core grammar. In general, these tokens were discarded.

A consequence of doing it this way is that the example you posted with a comment in the middle of a identifier, the identifier would not be a single identifier - this is the expected behaviour in all languages (from memory) that I've worked with.

The case of a comment within a string should be implicitly handled by the lexical analysis. The rules to handle a string have no interest in comments, and as such the comment is treated as the contents of the string. The same applies to a string (or quoted literal) within a comment - the string is a part of a comment, which is explicitly a single token; the rules for processing a comment have no interest in strings.

I hope that makes sense/helps.

Answer 5

It depends on what purpose your parser has. If you write a parser to build a parse tree for compiling than a comment has no semantic value beside potentially separating tokens (e.g method/comment/(/comment/)). In this case, its treated like spaces.

If your parser is part of a transpiler translating one source language into another source language or if your parser is a preprocessor taking a compilation unit in a source language, parsing it, modify it and writing the modified version back in the same source language, comments like anything else become very important.

Also if you have meta information in comments and you especially care for comments like when generating API-documentation like JavaDoc does, comments are all of a sudden very important.

Here comments often are attached to the tokens itself. If you find a comment you attach it to be a comment of a token. Since a token can have multiple tokens before and after, it is again purpose-depending how to handle those comments.

The idea of annotating non-comment tokens with having comments is to remove comments from the grammar altogether.

Once you have the parse tree some AST start to unpack comments representing each token by its own AST-Element but being attached to another AST-Element beside the usual contains-relationship. A good idea is to check all parser/AST implementations for source languages available in open-source IDE.

One very good implementation is the Eclipse compiler infrastructure for the Java language. They preserve comments during tokenization and represent comments within the AST - as far as I remember. Also, this parser/AST implementation preserves formatting.