8

Something that has long confused me is that so much software uses the terms "charset" and "encoding" as synonyms.

When people refer to a unicode "encoding", they always mean a ruleset for representing unicode characters as a sequence of bytes - like ASCII, or UTF-8. This seems reasonable and intuitive; the idea is that you are "encoding" those characters as bytes using the specified ruleset.

Since those rulesets sometimes only provide the ability to "encode" some subset of all unicode characters, you might imagine that a "charset" - short for 'set of characters' - would simply mean a set of unicode characters - without any regard for how those characters are encoded. An encoding would thus imply a charset (an encoding like ASCII, which only has rules for encoding 128 characters, would be associated with the charset of those 128 characters) but a charset need not imply an encoding (for example, UTF-8, UTF-16 and UTF-32 are all different encodings but can encode the same set of characters).

Yet - and here is the crux of my question - real-world usage of the word "charset" does not match what the construction of the word would imply. It is almost always used to mean "encoding".

For example:

How old is this curious (ab)use of language, and how did this counter-intuitive definition of 'charset' come to exist? Does it perhaps originate from a time when there truly was, in practice, a one-to-one mapping between encodings in use and sets of characters they supported? Or was there some particularly influential standard or specification that dictated this definition of the word?

  • 5
    The use of charset predates the Unicode standard, so while it might not have been a "one-to-one mapping", the characters required for a document tended to force the choice of encoding. But I don't have time to research the actual history of the word... – AmeliaBR Sep 7 '14 at 14:55
  • 2
    You talk about character sets as "a set of Unicode characters", as though someone started with Unicode and then picked subsets. It is more accurate to say many character sets existed prior to Unicode's invention (or at least prior to Unicode becoming ubiquitous), and Unicode is deliberately constructed to be a superset of them. – John Bartholomew Sep 7 '14 at 16:35
1

The concept of character sets is older than Unicode.

Before Unicode, a character set defined a set of characters and how each character was represented as bits. Most character sets mapped a character to a byte (which allowed a set of 256 characters), some mapped to two bytes, and a few (like ASCII) to only 7 bits. Different character sets often assigned different values to the same character, and there was no universal translation key between the various character sets in use.

Unicode was an attempt to solve this problem by unifying all the various character sets in a common "superset". For this purpose Unicode introduced some additional levels of abstraction, for example the concept of character encodings as something separate from the code point values. This allowed Unicode to redefine the pre-unicode character sets as unicode character encodings.

The charset attribute in HTML (which mirrors the charset parameter in the HTTP content-type header) for example, is from before unicode was widely adopted, but when it was decided to accept unicode as the universal character set of the internet, the charset attribute was just redefined to specify the encoding in use, but the name wasn't changed to allows backwards compatibility.

8

The term “charset” is of course short for “character set”, which in turn is an old term used for a character encoding, which can be interpreted as a) a mapping between sequences of bytes and characters, b) a mapping between integers (code numbers) and characters or c) a mapping between abstract characters and either integers or sequences of bytes.

“Character set” was taken into use since it is a short and simple expression. In it, “set” has a vague meaning (as the entire concept is vague), but it has later often been misinterpreted as a collection (“set” in set theoretical sense), as described in Dan Connelly’s (once) famous Character Set Considered Harmful.

What mattered most then was that the identifier charset was taken into use in MIME and in HTTP. HTTP 1.0 alias RFC 1945 quotes the MIME definition, “a method used with one or more tables to convert a sequence of octets into a sequence of characters” and then comments on the confusion: “This use of the term ‘character set’ is more commonly referred to as a ‘character encoding.’ However, since HTTP and MIME share the same registry, it is important that the terminology also be shared.”

1

RFC2045 introduced "character set" into MIME and by that time it was already clear not all encodings mapped one char to one octet. RFC2130 discusses this history a bit more.

Basically, "charset" was a more popular term when those standards were written, it's more succinct than "character encoding" or "text encoding scheme", and less ambiguous than "encoding".

  • It looks to me like the 'charset' parameter was already present in RFC 1341 (June 1992). – John Bartholomew Sep 7 '14 at 16:27
1

There are at least 2 encodings applicable to the transfer of text. The character encoding describes how characters (glyphs) are encoded. All languages use a subset of the UCS character set, which are often encoded to an 8 bit value. Standardized character sets have standard character encodings, and the terms are generally used interchangeably.

The transfer encoding describes how the text is transferred and is generally independent of the character encoding. Most transfer encodings are not transparent and alter the bytes sequence being transferred. 8-bit transfer encoding is transparent for 8-bit encodings. 7-bit transfer encoding can only transfer 7-bit character sets such as ASCII

ASCII uses 95 printable characters and a few more characters for carriage control (line feed, form feed, carriage return, backspace and tabs) encoded in 7 bits. There are are additional control characters using the remaining values. ASCII is a proper subset of the UCS character sets as well as many of the 8 bit character sets. ISO-8859-1 is also a proper subset of the UCS character set, but these characters have different encoding in UTF-8, which uses multiple bytes for character values greater than 127.

IBM mainframes use(d) the EBCDID which used a different 8-bit encoding. There are translation tables which map the printable characters between ASCII and ECBDIC character encodings. ECBDIC has a larger character set than ASCII because it has punctuation marks not available in ASCII. This may prevents full round trip transformation between these two encodings, if the initial encoding is ECBDIC.

These encoding were not sufficient for languages with characters not covered by the ASCII printable characters. Microsoft and others used 8-bit extensions to the ASCII character set and encoded additional characters with values above 127. A single extension was not sufficient for all languages as these extensions only added up to 96 characters to the character set. This resulted in multiple character sets (charsets) being used for different languages with different 8 bit character encoding. These character sets include the characters required for that language or language family. The additional characters may be encoded to different values in different charsets. English users are most likely to notice this with extended punctuation (left and right quotation marks, and dashes) which show up incorrectly if the wrong character set is used. This makes it important to know the character set used with an 8 bit character encoding.

For single byte encodings only characters in the character set can be encoded. Some languages required multi-byte encodings to cover their character set. The Unicode (UCS) character set requires multi-byte encodings for characters outside the ASCII character set. This character set is a superset of all the language specific encodings. UTF-8 is a compact encoding of the UCS character set. ASCII requires no additional bytes, and most (all?) European language characters can be encoded as one or two bytes. The full UCS-2 character set can be encoded in one to three bytes. All characters in the current UCS character set can be encoded in one to four bytes.

0

FWIW, IIRC, Back in the Commodore 64 days, a character set was a function from {0,...255} to the set of all uxv black and white images where u and v were maybe 8 and 12. So "changing your character set" meant using the POKE command in the BASIC language to modify the memory locations designated to hold those images. For example the ASCII code 65 meant A and the image of an A would be stored starting at memory address a*65+b for some b and I guess a=u*v.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.