When is it beneficial to use encodings other than UTF-8? Aside from dealing with pre-unicode documents, that is. And more importantly, why isn't UTF-8 the default in most languages? That is, why do I often need to explicitly set it?

  • 1
    Almost never. – user7043 Mar 31 '14 at 7:45
  • 2
    Quite often. And I really don't think this is a duplicate. – david.pfx Apr 2 '14 at 2:18
  • tell that to alllll the people who voted it to be a duplicate – Electric Coffee Apr 2 '14 at 13:54
  • @david.pfx: this is also too broad. – Martijn Pieters Apr 3 '14 at 7:22
  • @MartijnPieters: I know most of those people and I respect their opinions. While I wouldn't argue against closing the question, I'm still not sure it's that close a duplicate. – david.pfx Apr 3 '14 at 10:07

For an external encoding (i.e., an encoding of things not inside your program) it is very hard to beat UTF-8; it supports every character your users might ever reasonably need and there's lots of support in many OSes and tools. (The one place that counts as an exception to this is in file names, where you must use the platform's conventions if you want any kind of interoperability at all. Fortunately, many platforms now use UTF-8 for this so the warning is a moot point there.)

For an internal encoding, things are more complex. The issue is that a character in UTF-8 is not a constant number of bytes, which makes all sorts of operations rather more complex than you might hope. In particular, indexing into the string by character (a very common operation when doing string processing!) changes from an O(1) operation into an O(N) operation, and that can be a very significant performance issue. There are a number of possible workarounds, such as using a rope data-structure or converting the string into a fixed-width character format (typically ASCII, ISO 8859-1, UTF-16 or UTF-32, depending on the maximum Unicode value of the characters in the string). The problems that plague such formats (limited character support and/or endian-ness problems) don't actually apply here because you can only apply a transformation where it is meaningful and you are only using it as an internal encoding.

Don't think that you can get away with storing that internal encoding to disk or giving it to another program. It might be “convenient” but it's a problem waiting to happen; send/store the data as UTF-8.

And don't forget that there's a lot of legacy data out there, far too much to dismiss. Of particular concern are various East Asian languages which have complex encodings that are potentially quite a bit shorter than UTF-8, so resulting in less pressure to convert, but there are many other issues lurking even in Western systems. (I don't want to know what is happening in major bank databases…)

  • 4
    Depending on the kind of processing, even UTF-32 doesn't provide a fixed-width encoding. For example, accented characters in non-Latin scripts (and uncommon accented characters in Latin scripts) are represented by a sequence of multiple Unicode codepoints (multiple UTF-32 'characters'). – Bart van Ingen Schenau Mar 31 '14 at 9:27
  • 3
    -1 for the constant time access myth. Almost all string processing be done equally fast and convenient using any arbitrary unit (e.g. bytes or code units) for indexing and sequential iteration (from start and end). And as others point out, UTF-32 only gets you O(1) access to code points, but another important (I'd say more important) notion of character are grapheme clusters, and in that regard UTF-32 gets you nowhere. See also: utf8everywhere.org/#myths – user7043 Apr 1 '14 at 6:37
  • 2
    @david.pfx: The Zalgo text in the link from @RaphaelMiedl is an extreme example, but that was exactly what I was referring to.` – Bart van Ingen Schenau Apr 1 '14 at 6:40
  • 2
    @david.pfx: The meaning of 'character' is context dependent. Sometimes it means the same as codepoint, but sometimes it means the same as a grapheme cluster (that what most non-programmers would identify as a character in a displayed/printed text, i.e. a base character combined with all diacritical marks) – Bart van Ingen Schenau Apr 1 '14 at 10:53
  • 2
    @DonalFellows Could you give an example? Note that I encourage constant time indexing, but with byte indices rather than code point indices. What problem requires constant time access to the ith code point, yet not to the ith grapheme cluster, and can't make do with the nth byte? UTF-8 is no panacea for bad algorithms, but I don't know any problems where it prevents good algorithms. – user7043 Apr 1 '14 at 16:44

The answer is that UTF-8 is by far the best general-purpose data interchange encoding, and is almost mandatory if you are using any of the other protocols that build on it (mail, XML, HTML, etc).

However, UTF-8 is a multi-byte encoding and relatively new, so there are lots of situations where it is a poor choice. Here are a few.

  1. Internal encoding in Windows/C/C++/C#/Java/ObjectiveC. These environments do not internally support UTF-8 (or any multibyte encoding). Strings are respectively ANSI/UCS-2/UTF-16.

  2. Legacy code, especially C/C++. Strings are typically ANSI/ISO/UTF-16/UTF-32.

  3. Legacy data. There are vast mountains of textual data already encoded in some 8 bit format, including various code pages, JIS, etc.

The remaining cases involve the use of text files. They will likely remain an issue as long as plain old text files remain popular. The point is that text files do not encode their encoding, so the reader and writer have to make assumptions. Yes, there is something called a Byte Order Mark but it is neither required or recommended for UTF-8 files so any file containing 8 bit characters is of uncertain encoding.

Here are some examples using text files with little reason to allow or use UTF-8.

  1. Software tools. Things like sed, awk, tr etc may or may not work with UTF-8. It's often easier not to try.

  2. Compilers. Most computer languages are defined in terms of 7 bit ASCII and read plain text files from disk, with special tricks for extended characters.

  3. Log files, simple protocols, embedded systems. Sometimes 7/8 bit ASCII is just the easiest.

  4. Not always needed. Most European languages can be encoded in code page 850 or 1252, with possible savings in space and coded logic.

I confidently expect that many of these will go away over time, but they are real reasons to avoid UTF-8 in certain situations until then.

  • 1
    New? 1993 isn't exactly new. – whatsisname Mar 31 '14 at 22:36
  • 3
    @whatsisname: Relatively new. ASCII is 1963, code pages around 1981. The web only 50% in 2008. Plenty of older stuff out there. – david.pfx Apr 1 '14 at 0:05
  • I'm not sure what you're getting at with #3 and #5. Are you saying languages require source files to be 7 bit ASCII? Then that should be just another example of legacy systems/data/protocols. And for #5, unless you're referring to legacy systems, you should elaborate why it's "easiest". Also, #6 doesn't make sense to me. Even if it's not "needed" you can simplify everything, improve user experience, and rule out several kinds of data mangling by simply using UTF-8 whenever possible. That it's not needed should make no difference since using UTF-8 should be at least as easy, if not easier. – user7043 Apr 1 '14 at 6:45
  • @delnan: See edits. I know it won't satisfy everyone, but these are genuine answers to the question as put. – david.pfx Apr 1 '14 at 10:27
  • 1
    ASCII was being merrily misunderstood back in the early 1990's. Our printer thought # should render as £, which made C code rather “different” to read… – Donal Fellows Apr 1 '14 at 13:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.