Should character encodings besides UTF-8 (and maybe UTF-16/UTF-32) be deprecated?

Question

A pet peeve of mine is looking at so many software projects that have mountains of code for character set support. Don't get me wrong, I'm all for compatibility, and I'm happy that text editors let you open and save files in multiple character sets. What annoys me is how proliferation of non-universal character encodings is labeled “proper Unicode support” rather than “a problem”.

For example, let me pick on PostgreSQL and its character set support. PostgreSQL deals with two types of encodings:

• Client encoding: Used in communication between the client and the server.
• Server encoding: Used to store text internally in the database.

I can understand why supporting a lot of client encodings is a good thing. It enables clients that don't operate in UTF-8 to communicate with PostgreSQL without themselves needing to perform conversion. What I don't get is: why does PostgreSQL support multiple server encodings? Database files are (almost always) incompatible from one PostgreSQL version to the next, so cross-version compatibility is not the issue here.

UTF-8 is the only standard, ASCII-compatible character set that can encode all Unicode codepoints (if I'm wrong, let me know). I'm in the camp that UTF-8 is the best character set, but I am willing to put up with other universal character sets such as UTF-16 and UTF-32.

I believe all non-universal character sets should be deprecated. Is there any compelling reason they shouldn't?

Answer 1

Since you mentioned PostgreSQL, I can say with some authority that the main killer reason why non-UTF8 server-side encodings are supported in such detail is that the Japanese need it. Apparently, identical round-trip conversion between Unicode and the various Japanese "legacy" encodings is not always possible, and in some cases conversion tables are even different between vendors. It's baffling really, but it's apparently so. (The extensive character set support is also one of the reasons why PostgreSQL is so popular in Japan.)

Since we are talking about a database system, one of the main jobs is to be able to store and retrieve data reliably, as defined by the user, so lossy character set conversion sometimes won't fly. If you were dealing with the a web browser, say, where all that really matters is whether the result looks OK, then you could probably get away with supporting less encodings, but in a database system you have extra requirements.

Some of the other reasons mentioned in other answers also apply as supporting arguments. But as long as the Japanese veto it, character setup support cannot be reduced.

Answer 2

Two obvious reasons: depending on the data you're storing, converting to a different format could take quite a bit of time and extra room. If you're storing 400 megabytes of information, doubling the storage requirements is no big deal -- but if you're storing 400 terabytes it starts to mean a bit more. Converting 400 terabytes of data from (say) Shift-JIS to UTF-x could take a bit of time as well.

This becomes especially difficult if you have (for example) uptime guarantees that say the database will be available for all but, say, 10 minutes out of any given year, and you have a database that's being updated several hundred times a second. Mind you, it's still possible to manage major conversions in such a situation, but it's not something to be undertaken lightly. In some cases, it could easily take years of planning to get ready for such a conversion.

If you were starting with a database that (for example) only supported ASCII, there might be good reason to debate whether it made sense to add on support for all those encodings -- but if you already support them, there's little to gain from dropping support for them.

Note, in particular, that you'd probably gain next to nothing in the way of simplifying the code, or anything like that. They'd still need all the conversion routines to deal with conversions between client and server anyway. As such, dropping the support would mean dropping one (minor) function call in the "write to disk" and "read from disk" paths, but little (if anything else). If you supported even two encodings on disk, you wouldn't even gain that -- you'd still have the function call there, so all you'd really do would be restricting the range of encodings supported by that function.

At least if I was designing this, I'd probably write the core of the database to work in UCS-4, and then have conversion routines between the core and the disk, and between the core and the user. I'd use the same set of routines in both cases, so the simplest route would be to allow disk storage to use exactly the same set of encodings as clients were allowed to use.

Answer 3

There's a couple of problems with only storing UTF-8 on the server:

1. What is the limit of a VARCHAR(20) column? Is that 20 bytes, or 20 "characters" (and in Unicode, what's a "character" when you take combining characters, ligatures and so on into account?). Worse, what about CHAR(20) where it actually has to reserve the entire possible space: I believe in MySQL, it reserves 4 times the number of bytes for a UTF-8 encoded column (so 80 bytes for CHAR(20)) just to handle the worst-case.
2. You need to perform constant encoding conversions between the server encoding and your client encoding. You could argue that you want to stop supporting multiple client encodings as well, but unless you do that, then all strings need to be converted all the time. If you can match your server encoding and client encoding, then the conversions are not required.
3. As others have pointed out, UTF-8 is quite efficient for storing English text, but it's very inefficient for other languages - east Asian languages, in particular. You could allow the use UTF-16 or UTF-8 as suits, I suppose. Or compress text, but that makes indexing and searching inefficient.

Having said all that, I agree with you: legacy encodings are mostly pointless and Unicode is generally the best encoding to use for all new applications. If I were writing a database server from scratch today, I would only support Unicode and not support any legacy encoding at all.

The difference is that PostgreSQL and most other database servers in use today were around before Unicode was a viable option. So they already had support for legacy encodings (they weren't legacy back then, of course) and there's just not much point ripping all of that code out for largely ideological reasons.

Answer 4

Non-universal (and specifically single-byte) encodings do have their place: On systems that:

• Don't have enough memory to store the Unicode Character Database.
• Have a single-byte font hard-coded in ROM.
• Have no Internet access to provide a source of differently-encoded files.

That's true today for some types of embedded devices. But on the desktop, and in the server room, non-Unicode encodings should be long obsolete by now.

Answer 5

Unicode is fundamentally broken, and is unlikely to ever been fixed. It needs to be replaced by something better, something truly universal. If anything needs deprecating, it's Unicode.

Example issues with Unicide:

• UTF8 is a reasonable hack, but most UTF16 based software is broken. Most Windows apps that support Unicode use UTF16, including the OS itself. The most common issue is not supporting more than the basic plane, i.e. multi-word characters.

• Han unification is an unmitigated disaster. It's impossible to mix Japanese/Chinese/Korean text in a single document without extra metadata, and difficult to detect which font should be used.

• Combinational characters are another disaster. More sensible encoding schemes map one character to one code, which makes processing strings relatively sane. Unicode does not. Unicode isn't even consistent - Han characters are mostly combinations, but are not encoded as such, where as European combinational characters are.

• Some people's names can't be written correctly in Unicode, or are highly prone to being rendered incorrectly due to the issues mentioned above. This can have severe consequences, e.g. when trying to board aircraft with a passport that doesn't match what is (incorrectly) printed on the ticket.

Due to these issues and more, a lot of non-English software can't use Unicode and relies on local character encodings. This is particularly common with Japanese and Chinese software.

Ideally, Unicode should be deprecated. TRON character coding is a pretty good replacement for Unicode, and largely compatible for existing software that won't be updated.

Answer 6

UTF-8 is the best for you egocentric¹ English speaker. If you were Japanese, about 99% of your characters would take 3-4 bytes instead of two in UTF-16.

Non-latin dialects really suffer from UTF-8 on the size level. Don't forget that within a few years, most of your clients might be Chinese, and Chinese writing has millions of characters. You can't sustain that efficiently with UTF-8.

Otherwise, I hate it when I have text documents that are not in UTF-something. I'll often go out of my way if I need to to have proper encoding. In my book, non-Unicode encodings are dead.

_{1. Don't take the egocentric part personally. I wanted to make a colorful illustration and I don't really mean it.}

Answer 7

Maybe for writing, but not for reading.

There is a lot of existing content that uses those encodings, and some encodings like base64 aren't going anywhere because some text protocols mandate those as ways to embed binary data.

A real problem is auto-detection of encodings which leads to security holes. I wouldn't mind seeing some obscure encodings like UTF-7 just disappear.

Auto-detection also tends to deal badly with content produced by naively concatenating strings of bytes.

Answer 8

I can agree that the default character encoding for databases and new applications should be some sort of UTF variant. I personally would opt for UTF-16 since it seems to be a reasonable tradeoff on space and complexity (more so than UTF-8). That said, some character encodings still make sense in certain cases.

• If you are storing/transferring base64 text, you only need ASCII and you can even get away with 7-bit encoded protocols like email. The extra overhead of UTF-8 is unnecessary.
• Several files and existing data is built on these older character encodings, being able to read them is important.

Do note that there are 4 standard UTF normalization algorithms. If you are concerned about multi-codepoint characters, you can use one of the two normalization algorithms that collapse them into the equivalent single-codepoint character. The difference between them has to do with logical equivalence vs. physical equivalence of characters.