-1

When writing a tokenizer, what is the standard practice for handling aliased language keywords?

For example, notethat signed short int is a language keyword in C++ and several aliases might be allowed.

  • signed int
  • short int
  • signed
  • short
  • int

Do we map several different misspellings of a single language keyword to some canonical/standard value?

As a second example, in python we might want to provide support for the Spanish-language by aliasing the in keyword with en

for _0opcion en zero_opcions:
    for _1opcion en uno_opcions:
        for _2opcion en dos_opcions:
            print(param_pregunta.format(_0opcion, _1opcion, _2opcion))

2 Answers 2

5

Do we map several different misspellings of a single language keyword to some canonical/standard value?

Not in a tokenizer, no. A tokenizer or lexer is only turning a stream of characters into a stream of tokens.

The parser will usually then turn the tokens into a uniform internal representation for a signed short int.

3

In C++, the tokens signed, short, and int are keywords. The phrase signed short int is not a keyword, but a sequence of three tokens. The parser will later combine these into a type name, and even later the type checker will resolve this name to an actual type. A tokenizer doesn't have enough context to do that, just like the tokenizer cannot resolve template invocations or qualified names.

Of course, if you literally do want to provide multiple aliases for the same keyword, then the tokenizer could emit the same token for all of them. This would make it possible to support multiple spellings without having to modify the parser. However, error messages might become confusing. Consider a syntax error:

Keyword "in" not allowed here:
def width(en):
          ^^

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.