The range of characters is usually called a character set, or sometimes also a character repertoire or an abstract character repertoire. And there, you already have the problem: a set is unordered. It only tells you which characters exist, but it doesn't tell you how to represent that character on the disk or in a stream.
That's where the character encoding comes into play. The character encoding assigns a bit pattern to each character.
For example: the ASCII character set tells you that there exist characters such as A
or z
or 0
or Carriage Return. But that is all it tells you: that those characters exist.
The ASCII character encoding tells you that A
is encoded as the bit pattern 100 0001
, z
as 111 1010
, 0
as 011 0000
, and Carriage Return as 000 1101
.
In the Unicode world, things are a little bit different in that there is another abstraction in between: the codepoint. In Unicode, a character does not get assigned a concrete bit pattern, but an abstract number instead. This number is called the Unicode codepoint and is typically written in the form U+<hexadecimal representation of codepoint number>
. The mapping of numbers to abstract characters (i.e. a function from a subset of the natural numbers to the abstract character repertoire) is called a coded character set.
So, the Unicode encoding tells you that A
is encoded as the codepoint U+0041
, z
as U+007A
, 0
as U+0030
, Carriage Return as U+000D
, and ✅
as U+2705
.
However, this does not yet tell you how to store this codepoint on disk or transmit it over the network. That's what a UCS Transformation Format (UTF) (or more generally, a character encoding scheme) is for. Historically, there have been many of those, but nowadays, UTF-8 is absolutely dominant, with some residual use of UTF-16 and UTF-32.
The easiest UTF is UTF-32: While the Unicode Consortium decided in 2003 that it would limit codepoints to 21 bits, Unicode was originally designed with 32 bit codepoints, and the UTF-32 encoding is simply the binary representation of the codepoint. There is still one problem, though: filesystems and network protocols are typically octet-oriented, and a 32 bit character takes up 4 octets, therefore you need to decide whether you store it beginning with the least-significant or the most-significant octet. Actually, the Unicode Consortium decided that they will not decide, instead, there are actually two variants of UTF-32: UTF-32LE and UTF-32BE.
The biggest advantage of UTF-32 is that it is a fixed-length encoding. If you want to know how long a string is in characters, take the length in octets and divide by 4. If you want to know how much memory to allocate for a string in octets, take the length in characters and multiply by 4. If you want to read the 3rd character, read the 4 octets starting at address 3*4.
The biggest disadvantage is that it is incredibly wasteful. Since the Unicode Consortium decided in 2003 to only ever allocate up to 21 bit codepoints, 11 bits (about one third) of the UTF-32 encoding will always be 0
for every character. For characters in the Basic Multilingual Plane, at least 16 bits (50%) will always be 0
. For characters in the ASCII character set, 25 bits will always be 0
, which means that for pretty much any text in a "Western" script, up to 78% of all bits will be 0
.
UTF-16 is a bit more complex: characters in the Basic Multilingual Plane are encoded as a single 16 bit character which corresponds to the binary representation of the codepoint. Characters outside of the Basic Multilingual Plane are represented as a so-called surrogate pair of two 16 bit characters. So, UTF-16 is a variable-length encoding where a single Unicode character may be encoded as 1 or 2 16 bit characters (which in turn is 2 or 4 octets). For the same reason as above, there are two variants: UTF-16LE and UTF-16BE.
UTF-8 is an octet-based variable-length encoding. Being octet-based, the ordering is well-defined, therefore there are no UTF-8LE or UTF-8BE variants. UTF-8 has many advantages over all other encodings. Its only disadvantage is that it is variable-length, which means that you have to scan the entire string to figure out how long it is, and you can't index into the string without searching from the beginning. (In other words, indexing and computing the length are O(n).)
Other UTFs are only of historical interest: UTF-1, UTF-5, UTF-6, UTF-7, and UTF-8. UTF-9, and UTF-18 were only defined as jokes. UTF-7,5 was never accepted.
There are some variants that are in actual use: WTF-8, mUTF-8, CESU-8, and some others.