A pet peeve of mine is looking at so many software projects that have mountains of code for character set support. Don't get me wrong, I'm all for compatibility, and I'm happy that text editors let you open and save files in multiple character sets. What annoys me is how proliferation of non-universal character encodings is labeled “proper Unicode support” rather than “a problem”.

For example, let me pick on PostgreSQL and its character set support. PostgreSQL deals with two types of encodings:

  • Client encoding: Used in communication between the client and the server.
  • Server encoding: Used to store text internally in the database.

I can understand why supporting a lot of client encodings is a good thing. It enables clients that don't operate in UTF-8 to communicate with PostgreSQL without themselves needing to perform conversion. What I don't get is: why does PostgreSQL support multiple server encodings? Database files are (almost always) incompatible from one PostgreSQL version to the next, so cross-version compatibility is not the issue here.

UTF-8 is the only standard, ASCII-compatible character set that can encode all Unicode codepoints (if I'm wrong, let me know). I'm in the camp that UTF-8 is the best character set, but I am willing to put up with other universal character sets such as UTF-16 and UTF-32.

I believe all non-universal character sets should be deprecated. Is there any compelling reason they shouldn't?

  • 4
    @mario: The original definition of UTF-8 allowed up to 6 bytes. It was later artificially restricted to only cover the characters UTF-16 could support. – dan04 Jan 26 '11 at 5:48
  • 7
    At least PostgreSQL deliberately deals with multiple character encodings. It sucks to have to deal with a random mix of UTF-8 and windows-1252 because someone just didn't care. – dan04 Jan 26 '11 at 6:42
  • 5
    @dan04: Working with Russian texts used to be a pain, as they used multiple encodings that were substantially different and would usually just hack things to work by using different fonts (which would often lie about the encoding in use in their metadata). All in all, a horrible mess. I suspect they've cleaned up though – probably by moving to UTF-8 – because the number of support requests from that direction has dropped right off. – Donal Fellows Jan 26 '11 at 9:38
  • 3
    The theoretical Unicode range is from 0 to 0x10ffff. Nothing more. That's what the Unicode standard says. UTF-8 handles all of Unicode and always will. It doesn't cover the hypothetical range of an encoding that isn't Unicode, but it covers all of Unicode. – gnasher729 Jan 31 '15 at 12:34
  • 2
    utf8everywhere.org – phuclv Feb 1 '15 at 5:41

Since you mentioned PostgreSQL, I can say with some authority that the main killer reason why non-UTF8 server-side encodings are supported in such detail is that the Japanese need it. Apparently, identical round-trip conversion between Unicode and the various Japanese "legacy" encodings is not always possible, and in some cases conversion tables are even different between vendors. It's baffling really, but it's apparently so. (The extensive character set support is also one of the reasons why PostgreSQL is so popular in Japan.)

Since we are talking about a database system, one of the main jobs is to be able to store and retrieve data reliably, as defined by the user, so lossy character set conversion sometimes won't fly. If you were dealing with the a web browser, say, where all that really matters is whether the result looks OK, then you could probably get away with supporting less encodings, but in a database system you have extra requirements.

Some of the other reasons mentioned in other answers also apply as supporting arguments. But as long as the Japanese veto it, character setup support cannot be reduced.

  • So, because of these encodings, conversion of text to UTF-8 and back is lossy in general? Even if the conversion back is done immediately (rather than 6 months from now)? – Joey Adams Jul 15 '11 at 15:12
  • Joey Adams: Apparently so. – Peter Eisentraut Jul 17 '11 at 17:26
  • 3
    Google for “Han unification” to see why – Petr Viktorin Dec 8 '11 at 20:31

Two obvious reasons: depending on the data you're storing, converting to a different format could take quite a bit of time and extra room. If you're storing 400 megabytes of information, doubling the storage requirements is no big deal -- but if you're storing 400 terabytes it starts to mean a bit more. Converting 400 terabytes of data from (say) Shift-JIS to UTF-x could take a bit of time as well.

This becomes especially difficult if you have (for example) uptime guarantees that say the database will be available for all but, say, 10 minutes out of any given year, and you have a database that's being updated several hundred times a second. Mind you, it's still possible to manage major conversions in such a situation, but it's not something to be undertaken lightly. In some cases, it could easily take years of planning to get ready for such a conversion.

If you were starting with a database that (for example) only supported ASCII, there might be good reason to debate whether it made sense to add on support for all those encodings -- but if you already support them, there's little to gain from dropping support for them.

Note, in particular, that you'd probably gain next to nothing in the way of simplifying the code, or anything like that. They'd still need all the conversion routines to deal with conversions between client and server anyway. As such, dropping the support would mean dropping one (minor) function call in the "write to disk" and "read from disk" paths, but little (if anything else). If you supported even two encodings on disk, you wouldn't even gain that -- you'd still have the function call there, so all you'd really do would be restricting the range of encodings supported by that function.

At least if I was designing this, I'd probably write the core of the database to work in UCS-4, and then have conversion routines between the core and the disk, and between the core and the user. I'd use the same set of routines in both cases, so the simplest route would be to allow disk storage to use exactly the same set of encodings as clients were allowed to use.

  • 1
    Shift-JIS is non-self-synchronizing, which makes searching cumbersome. You would gain significant simplification by not supporting it. – dan04 Jan 26 '11 at 6:59
  • @dan04: if you already have time-proven search / indexing routines for Shift-JIS, switching to UTF-8 or even UCS2 would probably improve performance insignificantly. For a new database you might choose a better, more convenient and regular encoding, like UCS2 or UTF-16. – 9000 Jan 28 '11 at 2:22
  • @dan04: if you could get away with not supporting it at all, you'd gain quite a bit. As long as you support it coming from/going to clients, you're going to be stuck with most of its ugliness... – Jerry Coffin Jan 28 '11 at 2:25

There's a couple of problems with only storing UTF-8 on the server:

  1. What is the limit of a VARCHAR(20) column? Is that 20 bytes, or 20 "characters" (and in Unicode, what's a "character" when you take combining characters, ligatures and so on into account?). Worse, what about CHAR(20) where it actually has to reserve the entire possible space: I believe in MySQL, it reserves 4 times the number of bytes for a UTF-8 encoded column (so 80 bytes for CHAR(20)) just to handle the worst-case.
  2. You need to perform constant encoding conversions between the server encoding and your client encoding. You could argue that you want to stop supporting multiple client encodings as well, but unless you do that, then all strings need to be converted all the time. If you can match your server encoding and client encoding, then the conversions are not required.
  3. As others have pointed out, UTF-8 is quite efficient for storing English text, but it's very inefficient for other languages - east Asian languages, in particular. You could allow the use UTF-16 or UTF-8 as suits, I suppose. Or compress text, but that makes indexing and searching inefficient.

Having said all that, I agree with you: legacy encodings are mostly pointless and Unicode is generally the best encoding to use for all new applications. If I were writing a database server from scratch today, I would only support Unicode and not support any legacy encoding at all.

The difference is that PostgreSQL and most other database servers in use today were around before Unicode was a viable option. So they already had support for legacy encodings (they weren't legacy back then, of course) and there's just not much point ripping all of that code out for largely ideological reasons.

  • 11
    "but it's very inefficient for other languages - east Asian languages, in particular" Even in practice? Consider this Chinese Wikipedia page. Even though it displays an awful lot of Chinese characters, in the page source, ASCII characters overwhelm them almost 7:1. – Joey Adams Jan 26 '11 at 5:57
  • 2
    If the N in your CHAR(N) column is part of a well-defined identifier format (e.g., a VIN is defined to be exactly 17 characters), then it probably doesn't need combining characters or ligatures. If not, then N is just an arbitrary limit, which should be interpreted generously to avoid truncating data. – dan04 Jan 26 '11 at 6:37
  • 6
    @Joey Adams: that's true of HTML and XML where the markup itself makes up a large proportion of the text (and is why I think UTF-8 is a good choice for the web), but in a database you don't often store HTML. At the end of the day, it's only a factor of two (or less) difference, which is not that much really. – Dean Harding Jan 26 '11 at 7:55
  • 5
    Bullet point #2 in this answer is irrelevant: it applies whether or not Unicode is used. Bullet point #3 absolutely exaggerates the inefficiency and its scope. At the same time, this answer vastly understates the problems caused by legacy encodings. It is easy to assume the problem is not such a big deal if all you ever use in your life is English. – Timwi Jan 28 '11 at 4:25
  • 2
    @Dean: I didn’t know it wasn’t allowed to comment on an answer without posting one of my own. – Timwi Jan 31 '11 at 22:16

Non-universal (and specifically single-byte) encodings do have their place: On systems that:

  • Don't have enough memory to store the Unicode Character Database.
  • Have a single-byte font hard-coded in ROM.
  • Have no Internet access to provide a source of differently-encoded files.

That's true today for some types of embedded devices. But on the desktop, and in the server room, non-Unicode encodings should be long obsolete by now.

  • 4
    I used to have home computers like that. I got rid of most of them in the early 80s. – David Thornley Apr 15 '11 at 14:07

UTF-8 is the best for you egocentric1 English speaker. If you were Japanese, about 99% of your characters would take 3-4 bytes instead of two in UTF-16.

Non-latin dialects really suffer from UTF-8 on the size level. Don't forget that within a few years, most of your clients might be Chinese, and Chinese writing has millions of characters. You can't sustain that efficiently with UTF-8.

Otherwise, I hate it when I have text documents that are not in UTF-something. I'll often go out of my way if I need to to have proper encoding. In my book, non-Unicode encodings are dead.

1. Don't take the egocentric part personally. I wanted to make a colorful illustration and I don't really mean it.

  • 3
    @Matthew - 4x is clearly 4 times larger than x (for positive x). I don't see how asymptotic notation is relevant here. I've never seen a hard disk advertised with an asymptotic growth rate. Normally, the size stays the same throughout the life of the drive. – Steve314 Jan 26 '11 at 9:41
  • 3
    Millions of characters won't fit in Unicode anyway. According to the Wikipedia article, there's currently about sixty thousand Han characters. Since Unicode isn't just Chinese, that means that a fair number of Chinese characters will take four bytes in UTF-16, which is as long as UTF-8 gets nowadays. It would be interesting to see stats on lengths of Chinese texts in UTF-8 and UTF-16. – David Thornley Jan 26 '11 at 16:05
  • 6
    @David: >99% of all Japanese and Chinese writing uses characters that require only 2 bytes in UTF-16 and 3 in UTF-8. The characters that require more are very rare and/or historical. – Timwi Jan 28 '11 at 4:31
  • 10
    Keep in mind that Japanese and Chinese generally use fewer characters per word. I work with an app that has large language files in English, Japanese and Chinese, all encoded in utf-8. The Chinese file is actually the smallest, while the Japanese file is about 15% bigger than the English original. – Gort the Robot Apr 10 '14 at 18:16
  • 3
    Nonsense. Anything that takes two bytes in UTF-16 takes no more than 3 bytes in UTF-8. Anything that is four bytes in UTF-8 is 4 bytes in UTF-16. There are no "millions" of Chinese characters, and obviously they wouldn't fit into 16 bit. – gnasher729 Jan 31 '15 at 12:50

Unicode is fundamentally broken, and is unlikely to ever been fixed. It needs to be replaced by something better, something truly universal. If anything needs deprecating, it's Unicode.

Example issues with Unicide:

  • UTF8 is a reasonable hack, but most UTF16 based software is broken. Most Windows apps that support Unicode use UTF16, including the OS itself. The most common issue is not supporting more than the basic plane, i.e. multi-word characters.

  • Han unification is an unmitigated disaster. It's impossible to mix Japanese/Chinese/Korean text in a single document without extra metadata, and difficult to detect which font should be used.

  • Combinational characters are another disaster. More sensible encoding schemes map one character to one code, which makes processing strings relatively sane. Unicode does not. Unicode isn't even consistent - Han characters are mostly combinations, but are not encoded as such, where as European combinational characters are.

  • Some people's names can't be written correctly in Unicode, or are highly prone to being rendered incorrectly due to the issues mentioned above. This can have severe consequences, e.g. when trying to board aircraft with a passport that doesn't match what is (incorrectly) printed on the ticket.

Due to these issues and more, a lot of non-English software can't use Unicode and relies on local character encodings. This is particularly common with Japanese and Chinese software.

Ideally, Unicode should be deprecated. TRON character coding is a pretty good replacement for Unicode, and largely compatible for existing software that won't be updated.

  • Your claim that it is impossible to mix the different variants of characters (Japanese/Korean/Chinese) seems to be outdated since 15 years, the Unicode 3.2 standard at 2002. Unicode support Variation selectors, codepoints which after a han codepoint explicitly specify which form should be displayed. Also the combinatorial characters are specified both as "combining diacritical marks" with base characters (a°) and special glyphs (å), the process of converting them vice versa is "normalization". So, no, Unicode is not fundamentally broken. – Thorsten S. Mar 21 '18 at 16:32
  • 1
    You illustrate many of the flaws. Some languages use combinational characters, some don't, and Unicode can't decide which it prefers. As I pointed out, most software that claims to support Unicode doesn't understand those issues anyway and will display it wrong even with the selectors. Programmers should not be expected to be language experts, which is the other fundamental flaw in Unicode. – user Mar 22 '18 at 17:34

Maybe for writing, but not for reading.

There is a lot of existing content that uses those encodings, and some encodings like base64 aren't going anywhere because some text protocols mandate those as ways to embed binary data.

A real problem is auto-detection of encodings which leads to security holes. I wouldn't mind seeing some obscure encodings like UTF-7 just disappear.

Auto-detection also tends to deal badly with content produced by naively concatenating strings of bytes.

  • 7
    Base64 isn't a character encoding. – dan04 Jan 26 '11 at 7:08

I can agree that the default character encoding for databases and new applications should be some sort of UTF variant. I personally would opt for UTF-16 since it seems to be a reasonable tradeoff on space and complexity (more so than UTF-8). That said, some character encodings still make sense in certain cases.

  • If you are storing/transferring base64 text, you only need ASCII and you can even get away with 7-bit encoded protocols like email. The extra overhead of UTF-8 is unnecessary.
  • Several files and existing data is built on these older character encodings, being able to read them is important.

Do note that there are 4 standard UTF normalization algorithms. If you are concerned about multi-codepoint characters, you can use one of the two normalization algorithms that collapse them into the equivalent single-codepoint character. The difference between them has to do with logical equivalence vs. physical equivalence of characters.

  • 1
    Can downvoters please say why they downvoted? – Berin Loritsch Jan 28 '11 at 11:12
  • 3
    I didn't downvote, but the whole point of base64 is to transfer binary data down a text channel. If you could choose what encoding to use on that channel, you wouldn't use a text encoding at all. Even if your channel really is plain ASCII, base 64 is only using 6 out of 7 bits - a significant overhead already. – Steve314 Feb 1 '11 at 12:01
  • I hope someone didn't just read the bullet points. Those were the exceptions to using UTF. And you are incorrect about base 64 only using 6 out of 8 bytes. The first set of ASCII "characters" are non printable control characters, which forces some of the characters in base64 to use 7 of the 8 bytes. It purposely avoids the high bit because all those characters are not guaranteed to exist in every code page, while the characters from 0-127 are. – Berin Loritsch Feb 1 '11 at 13:13
  • 2
    @Berin - (1) no, but that "I agree" stuff isn't much without the bullet points, and (2) base 64 has 64 "digits". 64 digits is 6 bits worth, because 2^6==64. How you represent that in a 7 bit code-space (or 8 bits, or even 8 bytes if you must) is separate from how much data is actually there. Avoiding the non-printing characters etc is the reason for the overhead - it doesn't mean the overhead doesn't exist. Choose a channel designed for binary data and that overhead isn't there. – Steve314 Feb 1 '11 at 14:27
  • 3
    Bear in mind that base64 was invented to deal with sending binary data over a text-only channel. It is known to be inefficient (3:4 expansion), but deals with technical limitations in certain transport options. Legacy would be email and UseNet forums, but a more modern application would be embedding binary data in XML. Sometimes the proper channel does not exist, and you have to work through the limitations of the existing ones. – Berin Loritsch Feb 1 '11 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.