Are there historical problems with non-ASCII identifier characters in code?

Question

I frequently encounter recommendations to specifically keep to ASCII characters in field and function names in documentation, even though non-ASCII (modern Unicode) generally works perfectly. An example is the recent Python 3.10 documentation.

I've been a bad boy in the past and have successfully used non-ASCII characters in many projects, including in Java, C#, and Python 3. This started on personal projects maybe ten years ago. For most of my teams, this is generally beneficial, as we're not amateurs (everyone has some way of producing a λ, é, or a θ on their keyboard quickly, like a compose key or a switched keyboard layout) and it speeds communication on the team. I feel it's actually led to more readable code, not less.

What I'm led to wonder is whether there's a reason, substantiated in an actual project, for this rule; or if they're impulsively assuming that everyone is going to be copy-pasting those characters. Or, arguably worse, they're assuming that not all systems can handle basic Unicode in 2022; which I think has come down to the specific hardware we're working with and is typically a non-issue.

Can anyone cite any points in the past where non-UTF-8 has, objectively, caused a project to fail or hindered it in some way, and provide me with some context for this recommendation?

Answer 1

There are historical reasons for this guidance, that are mainly related to the lack of uniform encoding standard.

Encoding issues

Unicode dates back to the 90s'. Before it became mainstream, there was no global standard way to encode λ, é, or ä. The western character sets used to be limited to a single byte (octet to be more precise). This leaves room for only 128 characters on top of the ASCII ones. But there are many more accented local characters and greek letters than the 128 that were available.

This is why the ISO-8859 character sets were standardized in several variants. People used country-dependent settings. Suppose a German developper worked in ISO 8859-1 and had an ß in the identifier. This was encoded as 0xDF in the source file. If a greek developer opened the same source, using an ISO 8859-7 setting, the identifier would be displayed with an ί instead of ß. If the greek developer would type a ß it would lead to 0xE2 in the source file, which a French developer would then read as â. This was a total mess guaranteed. And when an US colleague looked at it, it just showed a dot, or a question mark, or, a semi-graphic character (after 1981, when extended ASCII became popular).

Tool support

The lack of uniformity triggered practical annoyances in text editors. I remember for example that the word-by-word move viewed non-ascii as a word separator. So the navigation was not as smooth as with ascii identifiers.

More seriously, there was a lacking tool support. I remember the first linker on MS-DOS had constraints about the length of the symbol identifiers and it was limited to ASCII character sets. This is btw not so old story according Wikipedia about GNU Compiler collection (see also this SO question):

Although the C++ language requires support for non-ASCII Unicode characters in identifiers, the feature has only been supported since GCC 10. As with the existing handling of string literals, the source file is assumed to be encoded in UTF-8. The feature is optional in C, but has been made available too since this change.

Momentum, prejudice and internationalization

To prevent all these nasty issues, a lot of coding standards pragmatically recommended the use of ASCII characters for identifiers. This creates some momentum.

Moreover there is a broad consensus for using English language in identifiers in the context of international projects, or open source projects looking for a broad community. Having this kind of expectations create some prejudice against opening up to unicode characters.

But not all projects have an international audience. There are lots of teams out there working in a local context and using native language in comments, in git commits and even in identifiers. A study of some 1.1 millions non-English git repos demonstrate that this is a large scale reality. Here unicode is relevant (after all, there must be a reason for new languages such as Swift, C# and others to accept unicode identifiers). Ironically, due to the historical problems, many keep using transliteration (e.g. in German ae instead of ä or, more ambiguous: in French e instead of é).

So, still nowadays, a significant number of people have kept in mind that there was something, and just continue to promote ASCII, although Unicode identifiers are now largely supported.

Edit: All unicode chars are still not equal!

Accepting unicode characters in an identifier does not ensure that all unicode characters are equally treated in all languages that claim to support it.

Unicode characters are categorized into classes, for example spacing, punctuation, letters (aka writing alphabet) and others. Some quick experimentation on my mac:

• Swift, C++ and C# interpret spacing characters properly as token separator, whereas python considers it as an error.
• Emojis are not letters. C++ and Swift accept them in an identifier, but python and C# don't (Example: t🙂w)
• Characters of class "letter" are generally accepted in an identifier in all my tests in the four languages. But some Egyptian hieroglyphs are not recognized as letter by C# (Examples: t𓀉w, t丳w, téw, the first being the problematic one)
• And good news for the mathematicians among us, π is a valid identifier in all these languages ;-)

Answer 2

The big problem is seeing two identifiers, and determining just by looking whether they are the same. Capital Latin, Cyrillic and Greek characters for example often look the same. Chinese characters I just couldn’t recognise. And I think Swift allows certain white space characters in identifiers, other languages might as well. Imagine having an identifier that is literally invisible.

Having to use a different keyboard would be insane. On Mac keyboards, there are a few dozen non-ASCII characters easily available (I can type them without thinking), so anything in this set I would be fine, anything outside I wouldn’t. In one case I defined a ≈≈ operator (== didn’t work for some reasons). Other OSs have different easily accessible characters.

So I’m not totally against it, but be reasonable.

@Greg, the compilers have absolutely no problem with unicode characters. It's the programmer. The compiler can easily distinguish between three different white space characters, or defining an m-dash or n-dash as an operator instead of the hyphen. It's me who has the problem.

PS. Unfortunately Swift doesn't support an operator named ² or ³, so you can't write y = x³ or y = (a+b)². Well, you can write the first one because x³ is a valid identifier, so you can write

let x = 5
let x² = x*x
let x³ = x²*x

Answer 3

Code style recommendations should always consider what kind of code the team is developing. If you are writing scientific formula-heavy code, using Greek letters and math symbols may make the code clearer. Some languages like the APL-family embrace this and use lots of such symbols. APL even comes with a special-purpose keyboard.

But for general-purpose application programming with a focus on maintainability, there are good reasons to disallow characters outside of the ASCII range. You mention some of the potential problems yourself.

• Almost all code guidelines emphasize meaningful and descriptive identifiers. Symbols like θ are only meaningful in certain specific domains, and it is only unambiguous if the whole codebase is in that domain.

• Problems with inputting characters. If you need a special keyboard or remember obscure key combinations to input characters it just makes everything more difficult. For specific domains, this may be a worthwhile investment, but in general, this is more trouble than it is worth.

• Many Unicode characters look alike. Historically there have been many problems with confusing O and 0, but with Unicode you get dozens of different characters which look exactly alike.

• Historically many languages and platforms did not support non-ASCII characters in identifiers. For example, I believe Python 2 did not support non-ASCII identifiers (although I may remember wrong since you claim this has worked for you?)

• Non-ASCII characters are associated with identifiers using non-English words. It is often mandated to only use English language in identifiers and comments since English is the lingua franca of software development.

Answer 4

Normalization issues

This is just a minor nuisance, but is something that you should be aware of. For many characters, Unicode has several valid representations that should compare equal.

For example in Python the following works:

Amélie = 5
print(Amélie)

but e.g. grep considers the identifiers different.

There are four different standard forms of Unicode normalization, and probably several non-standard versions that emit different glyphs in different formats. This can vary by editor and operating system. I know that in filenames Windows and Mac OS X use different normalization forms, but haven't tried how that works in editors.

Example situation where this could cause problems:

1. Several programmers write the code using different platforms. End result works fine, thanks to the programming language correctly implementing unicode comparison.

2. A security issue involving function vérification_de_sécurité() is discovered. The developer assigned to fixing it uses grep to search the whole codebase for uses of it, and finds several that get fixed.

3. Unknown to the developer, there is one case where the function name was typed on a different editor and had different underlying byte representation. It was not found by grep and remains unfixed.

Answer 5

everyone has some way of producing a λ, é, or a θ on their keyboard quickly, like a compose key or a switched keyboard layout

Not really. You have just added some difficulty, setup time or new starter config to your code.

If its a small team and you all know each other maybe its a bit of fun. But are you going to recommend it for a general case? Have you tested all the potential problems that might occur?

There is a difference between what works, and even what is good and what you can recommend. A Recommendation has to be easy to state, easy to follow and work in most generic situations that it might apply to. It might not be the absolute best for every situation, but it gets you past some common pitfalls or problem.

I can recommend you call your variable PI without worrying about your keyboard setup, editor, source control etc. Can you say the same about π?

Answer 6

There are problems, and they aren't just historical. Sticking to ASCII is not conservatism, it's prudence.

Unicode is huge and complex. Using Unicode (in languages and platforms that support it) is not much of a problem if you stick to a sufficiently tame subset. But you have to 1. define that subset and 2. enforce it. Enforcing a subset is not hard: you can run grep as part of your CI before accepting a submission. You just have to remember to do this. Picking a subset, though, runs into a lot of difficulties.

There are technical difficulties with Unicode. It's not enough to have an editor that supports UTF-8: that is, indeed, a safe assumption in 2022. You need development tools to support it as well, but for a given programming language, you'll know how good tool support is. (It's not just the compiler, it's also linkers, FFI, optimizers, static analyzers, debuggers, etc. In some cases, you may need stick to ASCII for symbols with external linkage.)

One of the things that make Unicode hard is there can be multiple ways to represent the same character. Your Unicode subset should be sufficiently restricted so that this doesn't happen. Are à and à the same identifier? One of them is LATIN SMALL LETTER A GRAVE and the other is LATIN SMALL LETTER A followed by COMBINING GRAVE ACCENT. (I'm not sure whether my browser, Stack Exchange and your browser will preserve the difference…) If your tools don't all agree on one, it can make life difficult. For example, code indexing tools had better apply the same normalization rules as the compiler, and everybody's editor had better apply the same rules as well.

Right-to-left text can make things interesting. Not all editors handle it well even in 2022. A recent paper Trojan Source: Invisible Vulnerabilities demonstrated several ways in which code can look like it says one thing, but actually be something completely different. Many of those ways rely on switching between right-to-left and left-to-right. Even allowing non-ASCII characters only in comments enables this kind of attack! Make sure not to allow text direction changes unless support for strings or comments in a right-to-left language is necessary.

In addition, here are a few of the difficulties in choosing a subset that's suitable for humans:

• Different people recognize different sets of characters. You have to know your audience. For example, Greek letters might be ok if your program is solely written by scientists, but I'd guess most programmers have a hard time telling a ξ from a ζ, let alone remember whether a variable was called 這 or 那.
• No font covers all of Unicode, and non-proportional fonts (which are generally necessary to read code without messing up indentation) tend to have less coverage. You need to pick characters that is in all the fonts used by anybody working on the project, because ���� is not a helpful identifier name.
• I have no idea how different screen readers cope with various Unicode characters. Do you? Or do you just decide that blind people will never be able to work effectively on your code?
• Many people have an easy way to input some non-ASCII characters, but which subset is easy depends on the locale settings and on operating system. This is less important than font coverage, because we spend a lot more time reading code than writing code, but on the other hand it's more limiting, because keyboard coverage is way worse than font coverage. For example, most keyboard layouts don't offer a way to input Greek letters.

Answer 7

There were definitely historical problems with using non-ASCII characters in toolchains until recently. It’s becoming more and more true, though, that UTF-8 with a byte-order mark will just work. In some newer languages, UTF-8 is the only format that’s ever been used for source files.

Some problems in addition to what’s already been mentioned:

For backward compatibility, many older tools will default to some 8-bit character set, and not understand UTF-8 input without special configuration. IBM’s mainframe compilers still default to a form of EBCDIC. Microsoft’s compilers defaulted to the current 8-bit code page until 2008, when they added the ability to auto-detect UTF-8 if and only if the file began with a byte order mark, and for several years afterward had no other way to enable it. Enough time has passed that this might no longer be a major concern.

The same string can often have multiple encodings in Unicode, which are supposed to display as identical. I’ve seen Python source that wrote variables like FooᐸBarᐳ, where ᐸ and ᐳ are Unified Canadian Aboriginal Syllabics, and therefore “alphanumeric” characters. But there are even different encodings for character that are supposed to be the same character. Other characters are completely invisible, or might make part of a string display in reverse.

Pragmatically, it can be difficult for other people to maintain code that uses characters they cannot type on their keyboards, although the right input method might solve this.