5

What limitations will we have if Unicode standards had decided to assign one and only one codepoint to every user-perceived character?

Currently, Unicode has code-points that correspond to combining characters. Such characters combine with a previous code-point (or sequence thereof) to present to the user what appears to be a single character.

From what I can see, the current standard is full of complexities. (We're not talking about encoding issue here. Even if we try to avoid any kind of complexities by using an encoding like UTF-32, the problem still persists.)

For example, in Unicode when a grapheme cluster is represented internally by a character sequence consisting of base character + accent, then when the user clicks the > button to skip to the next user-perceived character, we had to skip from the start of the base character to the end of the last character of the cluster.

Why does it need to be so hard? Why can't Unicode assign a single code point to every user perceived character such going to the user-perceived character is simply a matter of reading the next code point?

The unicode website seems to acknowledge this complexity:

In those relatively rare [not rare at all!] circumstances where programmers need to supply end users with user-perceived character counts, the counts should correspond to the number of segments delimited by grapheme clusters. Grapheme clusters may also be used in searching and matching

Diacritics are also the reason why things don't work as expected. For example, if I throw 2 ピ characters (japanese katakana PI using the unicode representation U+30d2 U+309a) into a String builder and reverse it, I would naturally expect the output to be 2 ピ characters (i.e. ピピ), but it gives an invalid output of ゚ピヒ !

If Unicode had assigned individual code points for each user-perceived character and scrapped the idea of grapheme clusters, this wouldn't have happened at all.

What I would like to know is, what exactly is wrong with representing every user-perceived character as one unicode codepoint? Is it likely that doing so would would exceed U+10FFFF possible code points (if it does exceed U+10FFFF code points I see no reason why they couldn't have set the limit to 2^32 in the first place), Even when there is so much spare space to include the whole family of Elf Languages ?

Unicode states:

If you wanted, for example, to give each “instance of a character on paper throughout history” its own code, you might need trillions or quadrillions of such codes;

But seriously that doesn't make sense at all. Where is the source for making such a claim? A trillion codes is quite an over-statement.

Improve this question
2
  • Welcome to the real world. Who knows the exact reasons why... perhaps it is true that some developers/designers over-complicate things(subconsciously even...?) just to give themselves more work. Small example - why does Windows use the "\" while Unix does "/" in directories? Don't get me started on SQL variants... I dunno, anyway you seem to be quite expert here already, I hope someone slightly more expert comes around!
    – Caffeinated
    Commented Dec 8, 2011 at 23:38
  • 4
    @Adel On why Windows uses backslash: blogs.msdn.com/b/larryosterman/archive/2005/06/24/432386.aspx
    – yannis
    Commented Dec 8, 2011 at 23:45

3 Answers 3

Reset to default
4

The origins of Unicode date back to 1987–1988, when Joseph D. Becker published the first Unicode draft proposal. A 16-bit fixed width encoding (UCS-2) was an indispensable part of his design. He was very proud of the distinction he made between characters and glyphs, just to squeeze everything to this small range. Unfortunately his naïve assumptions were proven wrong, or at least unrealistic, when in 1996 Unicode code-space was increased beyond the 16-bit limit and UCS-2 was extended to UTF-16. Currently Unicode has almost twice as much allocated codepoints than the code-space of Becker's original design.

[...] if it does exceed U+10FFFF code points I see no reason why they couldn't have set the limit to 2^32 in the first place [...]

It is currently limited by U+10FFFF because it's the maximum range possible to be encoded with UTF-16. UTF-16 is an ugly hack applied late in the day to extend the code-space. It's not extensible. UTF-8 on the other hand is. UTF-8 was originally allowed to encode 231 codepoints and was later limited to just match the same range as UTF-16. In fact you can extend the scheme used by UTF-8 indefinitely, even to encode 2128 codepoints and preserve all the nice properties of UTF-8 at the same time (lexicographically comparable, unambiguous substring matching with UTF-8 unaware code, etc...) except that you won't be able to determine the length of the codepoint from the first byte.

From what I can see, the current standard is full of complexities. Even if I try to avoid any kind of complexities by using an encoding like UTF-32, this problem still persists. It's not an encoding issue at all.

Exactly! I want to point out that variable length encodings are not the problem. Indeed, finding the next code-point in a UTF-8 string is simple, just scan for the next byte of the form 0xxxxxxx or 11xxxxxx. So we could easily increase the available codespace without increasing the average memory consumption. The real problem is that to find the next grapheme you must have a Unicode database available (in some form), which is definitely complex and costly for many simple applications.

Improve this answer
10
  • So in fact if we scraped the idea of UTF-16 and extend codepoint to 2^32, we could actually have enough space to incorporate all natural languages and a few art language ? Btw what's the solution to that real problem? Or what solutions can we use to attempt to solve that problem?
    – Pacerier
    Commented Dec 31, 2011 at 20:27
  • @Pacerier: one grapheme cluster may have arbitrary number of combining character, so no fixed-width codepoint per grapheme may do. What you may do though is 1) use a variable length encoding that encodes arbitrary length codepoints so that each represents a grapheme cluster or 2) leave the multiple codepoints per grapheme cluster used today but change the assignment of codepoints so you can determine the properties of codepoints by some simple rules (e.g. test some bit to determine if it's a base character).
    – ybungalobill
    Commented Dec 31, 2011 at 20:40
  • Also note that Unicode had more constraints. It took into consideration making conversion from existing encodings to Unicode simpler.
    – ybungalobill
    Commented Dec 31, 2011 at 20:45
  • The problem I have with unicode is that they are trying to do too many things. End up, they do things half-assedly. For example, unicode actually wants other people to tailor and invent algorithms for determining user-perceived-characters when that exactly is what they should have been doing. I've certainly not heard of anyone who actually did any tailoring for Unicode (unless its Apple/Google/MS etc). This is crazy, if all they did was KISSS and assigned one code point to every user-perceived character (which now I believe is possible having over a million
    – Pacerier
    Commented Feb 1, 2012 at 14:41
  • 1
    @Pacerier: some of the 'fun staff' is added to guarantee round trip conversion from existing encodings in use. But seriously, I don't quite understand how you envision code point assignment in which one code point would mean one grapheme cluster. I agree though that Unicode is overly complicated.
    – ybungalobill
    Commented Jun 22, 2012 at 11:06
3

If a script has M letters and N diacritics, then a precomposed representation requires up to M*N code points, while a decomposed representation requires only M+N. Combining characters prevent Unicode from "running out of code points".

Improve this answer
11
  • 2
    @Loki: Remember that Unicode originally planned to only have one "plane".
    – dan04
    Commented Dec 9, 2011 at 2:43
  • 1
    @Pacerier You really are missing the point. You cannot possibly have one code point per extended grapheme cluster, because the number of the latter easily exceeds 32 bits, let alone 21 of them. It just isn’t possible. You need to learn to iterating by grapheme clusters just as you need to learn how to iterate by words. Both are higher-level concepts of any open-ended problem space that is subject to infinite variation.
    – tchrist
    Commented May 13, 2012 at 14:24
  • 1
    @tchrist Iterating graphemes is starkly different than iterating words. In a language, there's no fixed number of possible word combinations, but there's certainly a fixed number of possible characters. As an example, Unicode allows us to merge e with ﾞ with ﾟ with ˆ with ˝. But the resulting character is unused in any language. It is unlikely that there are 2^32 (4 billion) standalone characters, because there's no such thing as language characters being subject to infinite variation.
    – Pacerier
    Commented May 13, 2012 at 15:41
  • 2
    @tchrist For example, in Japanese we can't simply combine a は + ﾞ + ﾟ because it doesn't make any sense in that language. Yes I'm aware Unicode allows us to combine a は + ﾞ + ﾟ, my argument is that that shouldn't have happened in the first place. Unicode should have just assigned a unique code point to each character for the ~7000 natural languages used on Earth.
    – Pacerier
    Commented May 13, 2012 at 16:59
  • 2
    @tchrist As in your example, in Greek, we can't simply combine a Ω + ´ + ` + ῾ because it doesn't belong to that language. I grant that you can combine Ω with ´ because it is valid in Greek. I grant that you can combine Ι + ´ + ¨ because it is valid as well. But combinations aren't arbitrary in a natural language. Even after adding all possible variations of diacritic combinations, Greek has only 188 characters. What's wrong with giving each of them a unique code point?
    – Pacerier
    Commented May 13, 2012 at 17:03
2

it's because back in the olden days a unit of data was 8 bits, the widespread 32-bit architecture is younger than the old ASCII standard on which the UTF variants are based (for backwards compatibility)

besides programmers on those old (sizeof(int)== 16 bits) architectures never liked using more resources than needed (memory in particular remember that UTF32 takes 4 times as much memory than UTF8 for text only in the latin alphabet) as back in the day you only got a few hundred kilobytes at your disposal at best

that and you don't need any other characters than those in ASCII if you are only programming for the english speaking (or any language with no accents)

one big lesson you can here learn is: anything that grows towards changing requirements is rarely future proof against the next change in requirements and generally only good enough for the current requirements and if someone thinks of a solution that exceeds the requirements it will likely get shot down for various reasons (most common will be performance, memory use and compatibility)

also allow me to point you towards this xkcd comic that might shed some light on the situation

Improve this answer
1
  • Yes but at times some standards are so horrible to live with it. Unicode has way way too many flaws. No one gets it right. Not even Microsoft nor Google. We should disband it for a simpler standard that actually has a chance of getting things right.
    – Pacerier
    Commented Dec 8, 2011 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.