10

I frequently encounter recommendations to specifically keep to ASCII characters in field and function names in documentation, even though non-ASCII (modern Unicode) generally works perfectly. An example is the recent Python 3.10 documentation.

I've been a bad boy in the past and have successfully used non-ASCII characters in many projects, including in Java, C#, and Python 3. This started on personal projects maybe ten years ago. For most of my teams, this is generally beneficial, as we're not amateurs (everyone has some way of producing a λ, é, or a θ on their keyboard quickly, like a compose key or a switched keyboard layout) and it speeds communication on the team. I feel it's actually led to more readable code, not less.

What I'm led to wonder is whether there's a reason, substantiated in an actual project, for this rule; or if they're impulsively assuming that everyone is going to be copy-pasting those characters. Or, arguably worse, they're assuming that not all systems can handle basic Unicode in 2022; which I think has come down to the specific hardware we're working with and is typically a non-issue.

Can anyone cite any points in the past where non-UTF-8 has, objectively, caused a project to fail or hindered it in some way, and provide me with some context for this recommendation?

12
  • 9
    In the Unicode character repertoire, there are many different letters and symbols which look the same. Traditionally the difference between O and 0 has been a source of bugs, but with Unicode this problem is much much bigger.
    – JacquesB
    Jan 29 at 16:40
  • 5
    Trojan Codes Jan 29 at 16:40
  • 1
    @PhilipKendall That was fascinating! I would love it if you found time to turn it into a proper answer. I never even considered that problem. Jan 29 at 23:29
  • 1
    Historically, most compilers didn't support any kind of Unicode input. Historically, most language specs didn't support identifier characters outside of A-Za-z0-9_ and maybe a couple other punctuation characters. So it wasn't a matter of style.
    – hobbs
    Jan 30 at 5:29
  • 1
    @MichaelMacha of course. But why would you open this can of worms to begin with?
    – DeepSpace
    Jan 30 at 14:49

7 Answers 7

13

There are historical reasons for this guidance, that are mainly related to the lack of uniform encoding standard.

Encoding issues

Unicode dates back to the 90s'. Before it became mainstream, there was no global standard way to encode λ, é, or ä. The western character sets used to be limited to a single byte (octet to be more precise). This leaves room for only 128 characters on top of the ASCII ones. But there are many more accented local characters and greek letters than the 128 that were available.

This is why the ISO-8859 character sets were standardized in several variants. People used country-dependent settings. Suppose a German developper worked in ISO 8859-1 and had an ß in the identifier. This was encoded as 0xDF in the source file. If a greek developer opened the same source, using an ISO 8859-7 setting, the identifier would be displayed with an ί instead of ß. If the greek developer would type a ß it would lead to 0xE2 in the source file, which a French developer would then read as â. This was a total mess guaranteed. And when an US colleague looked at it, it just showed a dot, or a question mark, or, a semi-graphic character (after 1981, when extended ASCII became popular).

Tool support

The lack of uniformity triggered practical annoyances in text editors. I remember for example that the word-by-word move viewed non-ascii as a word separator. So the navigation was not as smooth as with ascii identifiers.

More seriously, there was a lacking tool support. I remember the first linker on MS-DOS had constraints about the length of the symbol identifiers and it was limited to ASCII character sets. This is btw not so old story according Wikipedia about GNU Compiler collection (see also this SO question):

Although the C++ language requires support for non-ASCII Unicode characters in identifiers, the feature has only been supported since GCC 10. As with the existing handling of string literals, the source file is assumed to be encoded in UTF-8. The feature is optional in C, but has been made available too since this change.

Momentum, prejudice and internationalization

To prevent all these nasty issues, a lot of coding standards pragmatically recommended the use of ASCII characters for identifiers. This creates some momentum.

Moreover there is a broad consensus for using English language in identifiers in the context of international projects, or open source projects looking for a broad community. Having this kind of expectations create some prejudice against opening up to unicode characters.

But not all projects have an international audience. There are lots of teams out there working in a local context and using native language in comments, in git commits and even in identifiers. A study of some 1.1 millions non-English git repos demonstrate that this is a large scale reality. Here unicode is relevant (after all, there must be a reason for new languages such as Swift, C# and others to accept unicode identifiers). Ironically, due to the historical problems, many keep using transliteration (e.g. in German ae instead of ä or, more ambiguous: in French e instead of é).

So, still nowadays, a significant number of people have kept in mind that there was something, and just continue to promote ASCII, although Unicode identifiers are now largely supported.

Edit: All unicode chars are still not equal!

Accepting unicode characters in an identifier does not ensure that all unicode characters are equally treated in all languages that claim to support it.

Unicode characters are categorized into classes, for example spacing, punctuation, letters (aka writing alphabet) and others. Some quick experimentation on my mac:

  • Swift, C++ and C# interpret spacing characters properly as token separator, whereas python considers it as an error.
  • Emojis are not letters. C++ and Swift accept them in an identifier, but python and C# don't (Example: t🙂w)
  • Characters of class "letter" are generally accepted in an identifier in all my tests in the four languages. But some Egyptian hieroglyphs are not recognized as letter by C# (Examples: t𓀉w, t丳w, téw, the first being the problematic one)
  • And good news for the mathematicians among us, π is a valid identifier in all these languages ;-)
6
  • 2
    This is the answer. A lack of a standard global character encoding reduces us to the a subset supported by anyone using a western-style alphabet. Source code files must use unicode to support anything else. See C# identifiers. Jan 29 at 22:25
  • And I see you edited your answer just as I wrote my comment 🙂 Jan 29 at 22:27
  • @GregBurghardt Indeed, same timing! Isn't it nice to see that the same ideas come to our minds at the same time ;-)
    – Christophe
    Jan 29 at 22:42
  • that was true in the past. But today I'm pretty sure that unicode will work in all tools
    – Ewan
    Jan 30 at 17:47
  • 1
    @Erwan Bonus: thanks to your question, I added a section about persisting issues still today: I added a section about the non uniform manner that different languages handle different kind of unicode characters in identifiers.
    – Christophe
    Jan 30 at 23:31
7

The big problem is seeing two identifiers, and determining just by looking whether they are the same. Capital Latin, Cyrillic and Greek characters for example often look the same. Chinese characters I just couldn’t recognise. And I think Swift allows certain white space characters in identifiers, other languages might as well. Imagine having an identifier that is literally invisible.

Having to use a different keyboard would be insane. On Mac keyboards, there are a few dozen non-ASCII characters easily available (I can type them without thinking), so anything in this set I would be fine, anything outside I wouldn’t. In one case I defined a ≈≈ operator (== didn’t work for some reasons). Other OSs have different easily accessible characters.

So I’m not totally against it, but be reasonable.

@Greg, the compilers have absolutely no problem with unicode characters. It's the programmer. The compiler can easily distinguish between three different white space characters, or defining an m-dash or n-dash as an operator instead of the hyphen. It's me who has the problem.

PS. Unfortunately Swift doesn't support an operator named ² or ³, so you can't write y = x³ or y = (a+b)². Well, you can write the first one because x³ is a valid identifier, so you can write

let x = 5
let x² = x*x
let x³ = x²*x
5
  • 2
    I really think compilers and language design should eliminate the issue with invisible characters. Jan 29 at 22:16
  • @GregBurghardt If only! We'll return to that in another ten years. Jan 29 at 23:20
  • I am in total agreement about practicality, and I didn't even think about doppelganger characters. My interests are generally for scientific notation and concise code. (Doppelgangers would be a very funny thing to include in a password, though...) Jan 29 at 23:21
  • @GregBurghardt It's not just invisible characters: apparently it is possible to misuse mixed scripting (left to right and right to left alphabets) to construct invisible code made of visible characters: krebsonsecurity.com/2021/11/…
    – Christophe
    Jan 29 at 23:29
  • 4
    @GregBurghardt Even ignoring invisible characters and RTL/LTR issues, there are other readability problems. A, Α, and A are three different characters, treated by compilers (that support Unicode properly) correctly, but are indistinguishable from each other in almost every font you could name, and thus you need to know which one is being used in each instance to understand code that mixes them properly. The same issue exists to varying degrees with many characters in the standard Latin alphabet (for example, H, Η, and Н, which even have different associated sounds in the languages using them). Jan 30 at 2:54
6

Code style recommendations should always consider what kind of code the team is developing. If you are writing scientific formula-heavy code, using Greek letters and math symbols may make the code clearer. Some languages like the APL-family embrace this and use lots of such symbols. APL even comes with a special-purpose keyboard.

But for general-purpose application programming with a focus on maintainability, there are good reasons to disallow characters outside of the ASCII range. You mention some of the potential problems yourself.

  • Almost all code guidelines emphasize meaningful and descriptive identifiers. Symbols like θ are only meaningful in certain specific domains, and it is only unambiguous if the whole codebase is in that domain.

  • Problems with inputting characters. If you need a special keyboard or remember obscure key combinations to input characters it just makes everything more difficult. For specific domains, this may be a worthwhile investment, but in general, this is more trouble than it is worth.

  • Many Unicode characters look alike. Historically there have been many problems with confusing O and 0, but with Unicode you get dozens of different characters which look exactly alike.

  • Historically many languages and platforms did not support non-ASCII characters in identifiers. For example, I believe Python 2 did not support non-ASCII identifiers (although I may remember wrong since you claim this has worked for you?)

  • Non-ASCII characters are associated with identifiers using non-English words. It is often mandated to only use English language in identifiers and comments since English is the lingua franca of software development.

4
  • 2
    I'm not sure if typing ã is a big deal for some people. If their language uses it, they may have different keyboards than we have in the USA. Jan 29 at 20:42
  • I've definitely used Unicode identifiers (a) in teaching contexts for people who don't speak English natively, (b) in domain specific contexts where ASCII transliterations would obfuscate the code – sometimes, making the code look like a typeset formula is more important than being able to type or change the code. But that's pretty rare.
    – amon
    Jan 29 at 21:18
  • Doppelganger characters are really standing out for me here as a potential problem; I'm generally talking about going from a formula on a whiteboard to executable code. I might be wrong about having used it in Python 2, I haven't used it in quite some time. (Ideally, I never will again...) but yeah, speaking the same cultural context is definitely critical for a lot of different reasons. Jan 29 at 23:25
  • 1
    @GregBurghardt: Sure, but then you are limited to developers having that native language and keyboard. You can't easily use international developers or outsource. This just limits your options. Code style recommendation is usually there to reduce friction and risk, and this is an avoidable problem.
    – JacquesB
    Jan 30 at 16:50
5

Normalization issues

This is just a minor nuisance, but is something that you should be aware of. For many characters, Unicode has several valid representations that should compare equal.

For example in Python the following works:

Amélie = 5
print(Amélie)

but e.g. grep considers the identifiers different.

There are four different standard forms of Unicode normalization, and probably several non-standard versions that emit different glyphs in different formats. This can vary by editor and operating system. I know that in filenames Windows and Mac OS X use different normalization forms, but haven't tried how that works in editors.

Example situation where this could cause problems:

  1. Several programmers write the code using different platforms. End result works fine, thanks to the programming language correctly implementing unicode comparison.

  2. A security issue involving function vérification_de_sécurité() is discovered. The developer assigned to fixing it uses grep to search the whole codebase for uses of it, and finds several that get fixed.

  3. Unknown to the developer, there is one case where the function name was typed on a different editor and had different underlying byte representation. It was not found by grep and remains unfixed.

3
  • 2
    Would be great if all tools followed the Unicode rules correctly. I think publishing a fast and correct and fast algorithm to implement .equals(), .hasprefix() and .hassuffix() would be nice.
    – gnasher729
    Jan 30 at 12:44
  • 1
    @gnasher729 Unfortunately the rules also change over time as new versions of Unicode standards get published.
    – jpa
    Jan 30 at 13:30
  • 1
    For example, the MacOS file system states explicitly which Unicode version is used to compare filenames. (The reason: What is today an "unassigned character" might next year be "letter z with cedilla" as a random example. But you don't want to change whether two files stored on your hard drive have the same or different names, so when. that character gets defined, it will be treated as different in the file system).
    – gnasher729
    Jan 30 at 13:38
3

everyone has some way of producing a λ, é, or a θ on their keyboard quickly, like a compose key or a switched keyboard layout

Not really. You have just added some difficulty, setup time or new starter config to your code.

If its a small team and you all know each other maybe its a bit of fun. But are you going to recommend it for a general case? Have you tested all the potential problems that might occur?

There is a difference between what works, and even what is good and what you can recommend. A Recommendation has to be easy to state, easy to follow and work in most generic situations that it might apply to. It might not be the absolute best for every situation, but it gets you past some common pitfalls or problem.

I can recommend you call your variable PI without worrying about your keyboard setup, editor, source control etc. Can you say the same about π?

6
  • 1
    Michael was talking about his own team. He knows better than we do whether they have trouble typing something or not. Jan 29 at 21:32
  • 1
    @KilianFoth Sure. But his team today may not be his team tomorrow. Jan 29 at 22:18
  • 2
    To switch keyboard layouts, I literally just hit both shifts and I'm typing in Ηλληνικά. If I want a compose key, something that is literally a check box on most *nix systems, I just hit my right Alt; but it can be whatever you want. While it requires a little extra software, it isn't terribly that hard to set up on Windows. If someone can't do that, even after an explanation (polite and geeky of course), I am at a point in my career in which I can opt not to work with them. If I'm writing scientific code, I prefer λ for wavelength over writing out lambda every time. It's clear and it's quick. Jan 29 at 23:19
  • 2
    @MichaelMacha but what if someone joins your team who is not Greek, and does not know the greek alphabet or keyboard layout? Even if they set up your recommended software they now need to learn where to find these keys on a keyboard that isn't labelled with them. I personally would figure out the 2 or 3 that matter and use alt+1234 style shortcuts to type them, except this will be much slower than typing lamda. Especially on laptops without number pads or mobile devices. A reason it's typically avoided is because generally you write code assuming it won't always be you maintaining it.
    – IronSean
    Jan 30 at 2:12
  • just tried both shifts and it didnt work. can you come over and setup my pc for me?
    – Ewan
    Jan 30 at 17:30
3

There are problems, and they aren't just historical. Sticking to ASCII is not conservatism, it's prudence.

Unicode is huge and complex. Using Unicode (in languages and platforms that support it) is not much of a problem if you stick to a sufficiently tame subset. But you have to 1. define that subset and 2. enforce it. Enforcing a subset is not hard: you can run grep as part of your CI before accepting a submission. You just have to remember to do this. Picking a subset, though, runs into a lot of difficulties.

There are technical difficulties with Unicode. It's not enough to have an editor that supports UTF-8: that is, indeed, a safe assumption in 2022. You need development tools to support it as well, but for a given programming language, you'll know how good tool support is. (It's not just the compiler, it's also linkers, FFI, optimizers, static analyzers, debuggers, etc. In some cases, you may need stick to ASCII for symbols with external linkage.)

One of the things that make Unicode hard is there can be multiple ways to represent the same character. Your Unicode subset should be sufficiently restricted so that this doesn't happen. Are à and à the same identifier? One of them is LATIN SMALL LETTER A GRAVE and the other is LATIN SMALL LETTER A followed by COMBINING GRAVE ACCENT. (I'm not sure whether my browser, Stack Exchange and your browser will preserve the difference…) If your tools don't all agree on one, it can make life difficult. For example, code indexing tools had better apply the same normalization rules as the compiler, and everybody's editor had better apply the same rules as well.

Right-to-left text can make things interesting. Not all editors handle it well even in 2022. A recent paper Trojan Source: Invisible Vulnerabilities demonstrated several ways in which code can look like it says one thing, but actually be something completely different. Many of those ways rely on switching between right-to-left and left-to-right. Even allowing non-ASCII characters only in comments enables this kind of attack! Make sure not to allow text direction changes unless support for strings or comments in a right-to-left language is necessary.

In addition, here are a few of the difficulties in choosing a subset that's suitable for humans:

  • Different people recognize different sets of characters. You have to know your audience. For example, Greek letters might be ok if your program is solely written by scientists, but I'd guess most programmers have a hard time telling a ξ from a ζ, let alone remember whether a variable was called 這 or 那.
  • No font covers all of Unicode, and non-proportional fonts (which are generally necessary to read code without messing up indentation) tend to have less coverage. You need to pick characters that is in all the fonts used by anybody working on the project, because ���� is not a helpful identifier name.
  • I have no idea how different screen readers cope with various Unicode characters. Do you? Or do you just decide that blind people will never be able to work effectively on your code?
  • Many people have an easy way to input some non-ASCII characters, but which subset is easy depends on the locale settings and on operating system. This is less important than font coverage, because we spend a lot more time reading code than writing code, but on the other hand it's more limiting, because keyboard coverage is way worse than font coverage. For example, most keyboard layouts don't offer a way to input Greek letters.
1

There were definitely historical problems with using non-ASCII characters in toolchains until recently. It’s becoming more and more true, though, that UTF-8 with a byte-order mark will just work. In some newer languages, UTF-8 is the only format that’s ever been used for source files.

Some problems in addition to what’s already been mentioned:

For backward compatibility, many older tools will default to some 8-bit character set, and not understand UTF-8 input without special configuration. IBM’s mainframe compilers still default to a form of EBCDIC. Microsoft’s compilers defaulted to the current 8-bit code page until 2008, when they added the ability to auto-detect UTF-8 if and only if the file began with a byte order mark, and for several years afterward had no other way to enable it. Enough time has passed that this might no longer be a major concern.

The same string can often have multiple encodings in Unicode, which are supposed to display as identical. I’ve seen Python source that wrote variables like FooᐸBarᐳ, where and are Unified Canadian Aboriginal Syllabics, and therefore “alphanumeric” characters. But there are even different encodings for character that are supposed to be the same character. Other characters are completely invisible, or might make part of a string display in reverse.

Pragmatically, it can be difficult for other people to maintain code that uses characters they cannot type on their keyboards, although the right input method might solve this.

Not the answer you're looking for? Browse other questions tagged or ask your own question.