How are comments generally treated in programming languages and markup? I am writing a parser for some custom markup language and want to follow the principle of least surprise, so I'm trying to determine the general convention.
For example, should a comment embedded within a token 'interfere' with the token or not? Generally, is something like:
Sys/* comment */tem.out.println()
valid?
Also, if the language is sensitive to new lines, and the comment spans the new line, should the new line be considered or not?
stuff stuff /* this is comment
this is still comment */more stuff
be treated as
stuff stuff more stuff
or
stuff stuff
more stuff
?
I know what a few specific languages do, nor am I looking for opinions, but am looking for whether or not: is there a general consensus what is generally expected by a mark up in regards to tokens and new lines?
My particular context is a wiki-like markup.