An enumeration is the classic/C-ish way to define tokens, but that doesn't make it extraordinarily good – precisely because it is difficult to keep track of associated values. A token might contain the value of a literal (e.g. number or string), and might also contain metadata such as character stream offsets, line number and column, ….
A sum type as in ML languages is vastly preferable: as compact as an enumeration + data field, but with proper type safety. Unfortunately C#, C++, and Java do not support proper sum types, so that we have to resort to a class hierarchy. This is not ideal (higher memory overhead, the class hierarchy can't be sealed), but it's usually better than the dynamic typing route where the token has a field for a nullable object.
Note that in the context of a token hierarchy of a parser, it is okay to downcast to specific token types. Alternatively, you can use the Visitor Pattern to match specific token types in a type-safe manner.
Nevertheless, I would generally advise against a deep token class hierarchy. Instead of introducing an abstract
Break class, you might want to instead add methods/properties to the base
Token type like
IsBreak => false (and override in the subclasses). I'd possibly diverge from this in two cases:
- for keyword tokens, as in this design each keyword will be its own class. It's much more comfortable if these keyword classes can have nearly empty bodies and just inherit any defaults from their base.
- for binary/unary operators, since a parser is often only interested in whether a token is a binary operator rather than deciding which binary operator it is, e.g. when doing operator precedence parsing.
One alternative I want to mention for completeness is scannerless parsing where you don't have explicit tokens at all. Instead, lexing/tokenization would be directly integrated into the parser. While many parser generators require the tokens to be parsed in advance, e.g. PEG, Marpa, or ad-hoc techniques like Recursive Descent do not.