6

For the designer of a language syntax, what are some reasons to choose a repeated delimiter to escape that delimiter, instead of having a separate escape character to escape that delimiter. A common example would be escaping the character ' in a string delimited by 's. Two common approaches to get the string It isn't. delimited by ' would be repeating the delimiter

'It isn''t.'

and using a different character for escaping the delimiter (in this case a \)

'It isn\'t.'

I haven't created a (non-toy) parser/syntax, so there are probably things I'm not considering. However, using repeating the delimiter seems like a strange choice to me. It seems the parser would need to recognize the ', but then conditionally undo that decision based on the next character. While this a fairly minor check, it seems strange to me given the alternative of just having a character that always escapes the next character. Yet, given how commonly chosen the repeated delimiter is, I'm guessing there's good reasons for that choice. Are there particularly good reasons for choosing repeated delimiters? Are the particularly good reasons not to?

(I apologize in advance for what is likely misuse of the words "syntax", "parser", etc. where there are likely more correct words when getting into these kinds of language details)

1
  • 5
    ‘’’twas the night before Christmas...’ Commented Nov 12, 2020 at 23:28

5 Answers 5

10

It seems the parser would need to recognize the ', but then conditionally undo that decision based on the next character. While this a fairly minor check, it seems strange to me given the alternative of just having a character that always escapes the next character.

Who watches the watchmen? Or, more on topic, who escapes the escape character?

So you've decided to use \ as your escape character. Great! Now we don't have to do that pesky old "double character" kind of escaping!

Wait, but what if I want to put a backslash in my text? That means using... \\... Damn, tricked again!


The logic remains roughly the same regardless of whether you use a constant escape character or a dynamic one (i.e. by using repetition), because there's always going to be at least one case where you're dealing with a double character, i.e. when you're escaping the escape character.

The logic isn't all that different.

Fixed escape character

  • Is this character my escape character? (1)
    • Yes - Is the next character on the approved list of escapable characters? (2)
      • Yes - Use the related escaped character
      • No - Throw error, invalid syntax (note: this is the default behavior but in principle you could just opt to use a backslash as a 'real' character here)
    • No - This is a normal character.

Dynamic escape character

  • Is this character on the approved list of escapable characters? (2)
    • Yes - Is the next character the same as the previous one? (1)
      • Yes - Use the related escaped character
      • No - Use the initial character for its unescaped purpose.
    • No - This is a normal character.

As you can see, it's pretty much exactly the same in either case. You always perform the same work:

  • Checking if a character is of a specific value, marked as (1)
  • Looking through a list of escapable characters, marked as (2)

The two checks just happen in a different order depending on whether the escape character is fixed or not, but it's the same work regardless.


In truth, there is a slight difference in performance. If you do the list-lookup first, which is the more expensive one of the two operations, you're going to be doing more list lookups than if you're first doing the exact value matching (because if that fails, you don't even need to do the subsequent list lookup).

However, the performance difference here is not the most important consideration. You're not going to be parsing that much text that frequently. Any opinion you have on readability is going to be a significantly more impactful weight on that scale.

Personally, I prefer the \ escaping, but I may be biased as a born and bred C# developer. I feel like it telegraphs its intentions more easily to a reader. But this may simply be because my eyes were trained to spot it and my brain has been conditioned to look for a fixed escape character.

4
  • Excellent answer! The double backslash is a great remark! On the dynamic escape character algorithm with the string literals, I think there is no “throw error”: if the character is not the same, the string is closed and the next character belongs to the next token (e.g statement separator, pascal-like # operator or any string expression operator). Of course, if there was a mismatch from the user and the next token would appear invalid, this might result in other syntax error message and some clever error analysis techniques would be required to relate those to a string issue.
    – Christophe
    Commented Nov 13, 2020 at 10:58
  • @Christophe: I actually already went back to fix that mistake, but it seems I edited the fixed algorithm instead of the dynamic one... oops! Fixed now.
    – Flater
    Commented Nov 13, 2020 at 11:02
  • Thanks! For your first paragraph, the \\ seemed different, because after you read the first \ you knew you were definitely going to try to escape the next character, which is different from what happens for the '' case. However, I think your following explanation clearly explains why they aren't very different anyway. Commented Nov 13, 2020 at 17:25
  • @ShianiaWhite: You don't definitively know that, since the next character may be something that's not escape-friendly, e.g. \?. Secondly, while the most common behavior is to throw a compiler error when such a case happens, in theory any compiler could simply choose to interpret the \ (before a non-escape-friendly character) as a literal \. The logic would be exactly the same as it would work for '' escaping. But regardless of whether you throw an error or revert to a literal \, you do not definitively know that you're going to escape a character when you see the initial \
    – Flater
    Commented Nov 16, 2020 at 9:51
4

It can simplify parsing.

'It isn''t.' can be trivially tokenised as two strings: 'It isn', 't.'.

As long as your language tracks whitespace abstractly then there is a qualitative difference between 'It isn''t.' and 'It isn' 't.'.

A lot of languages already take the second ('It isn' 't.') and concatenate the two strings using an implied juxtapostion operator to a string token containing: It isnt..

Taking this one step further and having the operator emit the quote character when there is no whitespace between characters is the next step. Given the logical string tokens: It isn, ', t. Which are catenated together.

Of course you don't have to reify the ' token, its just implied.

Contrast this with say \ and you have to do a lot of speciality parsing before making tokens. This makes the lexer much more complex. Whereas the complexity in the parser itself already exists.

However thats not the end of the story. I'm presuming the lack of need to escape character code sequence etc. In which case the story reverses. The lexer already has the machinery for substituting character sequences in, so why would you not make use of it for escaping the quote character?

1
  • I hadn't considered the case when you wouldn't escape other characters. And your explanation of why this then makes the repeated quoting a simpler case really helps for understanding why that might be the first solution. Commented Nov 13, 2020 at 17:23
4

The doubling of the delimiter in string literals is quite old but not so common anymore nowadays. The first languages that used this syntax were probably ALGOL and FORTRAN.

Quote stuffing -- as it is sometimes called -- has the advantage of simplicity for the user, and reducing the reserved characters in the string. For the language implemetner, it's easy to implement, since the parsing the literal requires to monitor quotes, and a quick look-up of the next character allows to decide if the quote is closing the literal of if the quote is to be used as a special character in the literal. Lastly, the error handling is simplified as well: with an escape character you have also to consider the case where the second character is not valid in relation with the escape.

However, this apparent simplicity comes with some complications when it comes to adding other non-printable characters: embedding tabs and CRLF was already a common need, embedding any ascii code was a luxury needed, but in the Unicode era, it is a must considering that many codes are not necessarily available on the keyboard. This feature is easy to add with an escape sequence parsing (e.g. \x or \u) . With quote stuffing, you'd have to add other means to achieve this end (for example, pascal's #nn between quote-enclosed substrings): this adds another kind of token to your parser, and make string litterals look less homogeneous.

1

It’s better to not require any fixed delimiter, as with Perl’s heredoc syntax or embedded resources in .net (C# 11 is adding raw string literals without a fixed delimiter). If you can’t do that then you need an escape mechanism if your delimiter can’t be outside of the acceptable range for your content.

Allowing doubling to escape the escape character has a long history (thus will be very familiar and easy to remember), requiring the escaped character to be encoded in a different format, i.e. JS’s \x5C, is more powerful, but the best option will be to do both.

0

Although it isn't as pretty as some other choices, you can use one character for delimiters and a different character to start an escape sequence. If you ensure that the escape sequence can never use the delimiter, you've got a simple-to-parse syntax. Your escape sequences only need to escape the escape character and the delimiter:

'Isn%27t this a 100%25 easy syntax?'  

So doubling is not required. However, doubling allows you to avoid using two special characters (e.g. using doubled 's to escape the ' delimiter) or allows more mnemonic escape sequences (\', \\, \n, etc).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.