2

Many programs will supply one or more of the following as file encoding formats: UTF-8, UTF-16, UTF-32 and simply Unicode. How do I know what Unicode Transformation Format Unicode is referring to? I'm assuming it correlates to to one of the 8/16/32 formats but I have not been able to find that on the Unicode website.

8
  • 3
    None of them, and all of them. Primarily it indicates that the program's authors know very little about Unicode. – user7043 Aug 2 '16 at 19:28
  • 4
    When Microsoft says Unicode, they generally mean little-endian UTF-16 (or UCS2). – CodesInChaos Aug 2 '16 at 19:30
  • 2
    @GovindRai Because Microsoft does so consistently. That's probably a historic accident. Microsoft introduced unicode at a time when UTF didn't exist yet and used the natural choice of UCS-2 (each codepoint encoded using a 16 bit word). – CodesInChaos Aug 2 '16 at 19:48
  • 3
    I agree with delnan's comment. The only thing you can reliably infer from that, is that the author of the software doesn't know what he is talking about, and you should probably not use it to process any non-7-bit-ASCII text. – Jörg W Mittag Aug 2 '16 at 20:02
  • 1
    @GovindRai Because they explicitly state what they mean in the system documentation for Unicode: "Unicode-enabled functions[...] use UTF-16 (wide character) encoding, which is the most common encoding of Unicode and the one used for native Unicode encoding on Windows operating systems." – Jules Aug 3 '16 at 16:43
4

Unicode is a standard that defines a list of abstract characters, rendering rules, code points, composition rules, and encoding systems. For example:

UTF-8, UTF-16 and UTF32 are encoding formats for unicode characters on 8,16, or 32 bit integers. It should be added that when looking at the byte streams (e.g. files), UTF-16 and UTF-32 have both a big endian (BE) and a little endian (LE) variant.

So speaking of unicode encoding is misleading: there is not a single one. But there are common usage of this term:

  • on Windows, when you say UNICODE, it is assumed that it's UTF-16 encoding. In the windows API, the default unicode encoding uses a wchar_t which is mapped to a 16 bit integer.
  • on Linux, the preferred encoding for unicode is UTF-8. There's by the way an UTF-8 everywhere manifest that explains the rationale behind and try to promote this practice.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.