6

(Not entirely sure whether this should go in the information-security StackExchange instead; feel free to move it there if that's where it belongs.)

Unicode has many, many instances of pairs or larger sets of characters with identical glyphs nevertheless being assigned to separate codepoints (for instance, the Latin capital letter A, the Cyrillic capital letter А, and the Greek capital letter Α all have identical glyphs, but are assigned to codepoints U+0041, U+0410, and U+0391, respectively). This causes severe security issues, as well as the more minor problem of cluttering up Unicode with redundant characters.

Why doesn't Unicode assign all characters that share a particular glyph to the same codepoint, which would resolve both of these problems?

5
  • 5
    If you were to convert the "glyph" A to lowercase what would you expect to get? a or α? Besides that you really ought to provide a link proving that the 3 codepoints actually are the same "glyph".
    – Bent
    Commented Apr 4, 2018 at 23:11
  • 2
    I dont think unicode species the glyph, consider all the different glyphs that pass for the letter "a". It is confusing when the customary glyphs for different code points look a lot alike.
    – ddyer
    Commented Apr 4, 2018 at 23:41
  • 2
    This kinda begs the question, if someone created a font that gave every codepoint glyphs with unique appearances would anyone be willing to use it? Commented Apr 5, 2018 at 0:31
  • Incidentally, a great April Fools prank to play on a fellow programmer is to replace the standard semicolon ; with the greek question mark character ;.
    – Neil
    Commented Apr 5, 2018 at 6:44
  • 1
    @Neil OK you stay away from my terminal. Commented Apr 27, 2018 at 18:42

3 Answers 3

23

The short answer to this question is, "Unicode encodes characters, not glyphs". But like many questions about Unicode, a related answer is "plain text may be plain, but it's not simple". A good place to start unpacking this question is chapter 1, Introduction, of The Unicode Standard (TUS).

You say:

Unicode has many, many instances of pairs or larger sets of characters with identical glyphs nevertheless being assigned to separate codepoints (for instance, the Latin capital letter A, the Cyrillic capital letter А, and the Greek capital letter Α all have identical glyphs, but are assigned to codepoints U+0041, U+0410, and U+0391, respectively).

It is a mistake to say "Unicode has …[characters]… with identical glyphs", because Unicode does not standardise glyphs.

The Unicode Standard specifies a numeric value (code point) and a name for each of its characters. [It also defines]… a character’s case, directionality, …alphabetic properties…, and other semantic values.…" (TUS, Chapter 1, p.1)

Notice that the concept of "glyph" does not appear in that list. Unicode does not encode glyphs.

The difference between identifying a character and rendering it on screen or paper is crucial to understanding the Unicode Standard’s role in text processing. The character identified by a Unicode code point is an abstract entity, such as “latin capital letter a” or “bengali digit five”. The mark made on screen or paper, called a glyph, is a visual representation of the character. The Unicode Standard does not define glyph images. That is, the standard defines how characters are interpreted, not how glyphs are rendered. (TUS, section 1.3 Characters and Glyphs, p. 6)

So what defines how glyphs are rendered? Fonts, and the text layout engine, and their character-to-glyph mappings. If you give me the three characters Latin capital letter A, and Cyrillic capital letter А, and Greek capital letter Α, I could choose three fonts to render them which made them look identical. I could choose three other fonts which made them look different. Unicode's architecture has no say there.

Characters that look similar, or homographs, are a problem even within character sets for a single writing system. I can present you with a domain name "PaypaI.com", which will look pretty similar to "Paypal.com", as long as I can choose a font where uppercase "I" looks identical to lower-case "l". In some fonts, digit "0" and uppercase letter "O" look darn similar. Homographs abound.

It gets worse. Consider U+0020 " " and U+00A0 " ". In just about every reasonable font, they will have identical glyphs. But they are distinct characters in many character sets, including Unicode, ISO 8859-1, and Windows CP-1252 "ANSI". Why? Because the first is a conventional space character, and the second is a non-breaking space. They have identical appearance, but have properties which affect text layout differently.

You say:

This causes severe security issues

I challenge your word "severe". Unicode makes many things possible which were difficult before, including some homograph attacks. But as the "PaypaI"/"Paypal" example shows, homograph attacks exist even within a single script. And within the landscape of security risks caused by text representations of domain names and the like, I argue that the attacks Unicode makes easier are medium to minor. Links where innocuous link text points to a brazenly malicious URL are a bigger problem. Users who click on such links are a bigger problem. Users who misinterpret long domain names like friendlybank.com.security_department.malicious.example.com, thinking it says friendlybank.com/security_department/, is a bigger problem.

I also challenge your word "causes". All non-trivial design involves tradeoffs. Unicode opens the door to new benefits, traded off with new risks. The designers of Unicode take the risks seriously: see UTR#36 Unicode Security Considerations.

To someone who is well-served by Latin-script text, the benefit of Bengali support may not carry much weight, and the novel homograph risks might be dazzling. But to the next billion internet users, who will largely prefer non-Latin scripts, the language coverage of Unicode is indispensable, and well worth the tradeoff of some risks.

Turning to "redundancy", you say:

…the more minor problem of cluttering up Unicode with redundant characters.

The three examples you give, U+0041 LATIN CAPITAL LETTER A, U+0410 CYRILLIC CAPITAL LETTER A, and U+0391 GREEK CAPITAL LETTER ALPHA, are not redundant for Unicode's text processing objectives in at least two ways.

The first way is case mapping. Latin, Cyrillic, and Greek scripts all have the notion of upper-case and lower-case versions of letters in a 1:1 relationship. What is the upper-case counterpart to U+0061 LATIN SMALL LETTER A? U+0041, obviously. What is the upper-case counterpart to U+0430 CYRILLIC SMALL LETTER A? U+03B1 GREEK SMALL LETTER ALPHA? To have one codepoint for capital letter A that mapped to one of three different small letters, depending on whether the text is Latin, Cyrillic, or Greek script, would be an implementation nightmare. It is much more efficient to have separate codepoints for each capital letter A, that encodes their different text properties.

The second way is round-trip convertibility with legacy encodings. The standard says,

Character identity is preserved for interchange with a number of different base standards, including national, international, and vendor standards. … This choice guarantees the existence of a mapping between the Unicode Standard and base standards. Accurate convertibility is guaranteed between the Unicode Standard and other standards in wide usage as of May 1993.… (TUS, section 2.2 Unicode Design Principles, subsection Convertibility, p. 23-24)

One of those "other standards" is ISO/IEC_8859-5, which encodes Cyrillic characters including 0xB0 Cyrillic Capital Letter A, and an ASCII-compatible block with 0x41 Latin Capital Letter A. Convertibility requires that Unicode have separate codepoints for these two characters. Another "other standard" is ISO/IEC_8859-7, which encodes Greek characters including, you guessed it, 0xC1 Greek Capital Letter Alpha, as well as ASCII-compatible 0x41 Latin Capital Letter A. Convertibility also requires separate codepoints for these two characters. And so, convertibility as well as case-mapping justifies Unicode having three distinct codepoints for these three characters.

Finally, you ask:

Why doesn't Unicode assign all characters that share a particular glyph to the same codepoint, which would resolve both of these problems [homograph attacks and "redundancy"]?

Because that would make it harder for Unicode to reach its goals of a "universal", "efficient", "unambiguous" text encoding (TUS, Section 1.2 Design Goals, p. 4). Because that would not "resolve" either security issues in general, or homograph attacks in particular; they would still be plentiful. Because the "redundancy" you see is not really a problem for text systems actually using Unicode to represent the world's plain text information.

[Update: added paragraphs on case-mapping and convertibility, as reminded by Nicol Bolas's answer. Thanks for the reminder.]

2
  • 1
    There are plenty of fonts where rn and m look identical or almost identical.
    – gnasher729
    Commented Nov 11, 2018 at 0:53
  • @gnasher729 I take it you are amplifying the point I made, "Characters that look similar, or homographs, are a problem even within character sets for a single writing system." And it turns out that in the font used by StackExchange and my web browser, your example is on point: "rn" (lowercase 'r' and 'n') does look identical to "m" (lowercase 'm'). Commented Nov 12, 2018 at 5:40
8

Consider U+0391: GREEK CAPITAL LETTER ALPHA. The lowercase version of that is U+03B1: GREEK SMALL LETTER ALPHA. If this were instead U+0041, the lowercase version would be U+0061, which is very much a different glyph from U+03B1.

Now you might say that the Cyrilic upper/lowercase versions both use the same glyph as the Latin upper/lowercase versions. But the Cyrilic block was adopted from a particular encoding: ISO/IEC 8859-5: Latin/Cyrilic alphabet. Most of the Cyrilic letters have no equivalent in the Latin alphabet, but there are several in that standard which do. In order to allow Unicode to properly round-trip encode text that was encoded in 8859-5, it had to retain those Cyrilic characters that shared glyphs with Latin or other characters.

It should also be noted that most Cyrilic letters that share glyphs with Latin letters have different names. And therefore, if you wanted to read Cyrilic text letter-by-letter, you need to be able to tell the difference between the Latin and Cyrilic letters.

In short, these encodings are not redundant. Unicode is about the character, not its visual representation. Overall, Unicode was never meant to have a unique mapping from glyph to codepoint.

3
  • I disagree. Unicode wanted very much to have a stable mapping from glyph to canonical code point. In nearly all cases where this isn't so, it's a compromise for backward compatibility, in particular round-trip compatibility (the ability to encode from legacy to unicode and back without information loss). This means that many characters which already had different code points in different character sets which were adopted into Unicode wholesale now continue to have multiple code points. But that is not what the consortium wanted; it was always a compromise to get Uniode adopted at all. Commented Apr 5, 2018 at 6:08
  • @KilianFoth: "have a stable mapping from glyph to canonical code point" Unicode already has a "stable mapping". What it doesn't have is a necessarily unique mapping. And how does this explain Greek Capital Letter A using the same glyph as the Latin version, but having different lowercase glyphs? Commented Apr 5, 2018 at 6:55
  • @KilianFoth I think it's incorrect to say Unicode wanted a "mapping from glyph to canonical code point". See my answer for more on "Unicode encodes characters, not glyphs". Maybe what you meant to say is, Unicode wanted very much to have an unique code point for each character. But still, The Unicode Standard clearly says that the behaviour of fonts and text renderers, even when they choose to render multiple characters with the same glyph, are beyond the scope of Unicode. Commented Apr 5, 2018 at 7:27
2

In my opinion, the main principle behind Unicode is not achieving some kind of linguistic purity, but being useful in the real world:

  1. If things don't get their own code point in Unicode, they cannot be distinguished without additional metadata. This may be exactly what one wants in one case (think of the glyph variants for lower case Latin a), but not in another (e.g. Latin vs Cyrillic "a").

  2. A character set such as ASCII does not just encode individual characters in random fashion but reflects alphabetical order and makes uppercasing/lowercasing comparatively easy to implement. While these features are not strictly necessary for encoding text, it certainly makes life easier for developers and the computer.

  3. Unicode does minimize the amount of "identical" characters where this is sensible, a look at CJK Unified Ideographs illustrates this well. Chinese, Japanese and Korean all use Chinese Characters to some degree.

    A huge subset of these characters (tens of thousands) can be considered identical across languages (just like Latin a is shared across many languages) and therefore only get a single code point each. Some glyphs may nevertheless slightly differ in appearance (see the article), depending on the display language/font.

    On the other hand, there is also a substantial amount of "simplified" and variant characters which are synonymous in meaning to another character but receive their own code point in order to encode simplified and traditional Chinese with no additional metadata (e.g. markup).

  4. There is an inherent element of arbitrariness to all this. Consider the Antiqua vs. Fraktur dispute. If people want to display/print German in Blackletter then they have to select a suitable font in a rich text document, Unicode does not consider Blackletter as distinct characters... except when it does(!) in the capacity as mathematical symbols.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.