I am wondering why unicode encoding is necessary in JavaScript. I am looking at utf8.js as an example. I am also looking at the utf8 spec, but am not really following the different pieces of data. Also, I don't fully understand when and where you are supposed to encode/decode, and what the "current" format is of some set of bytes in a programming language, so that's making it difficult to follow this.

UTF-8’s encoder’s handler, given a stream and code point, runs these steps:...

I would like to know what the stream is, and what the code point is. I understand that code points are characters, which in utf8 might be composed of multiple 8-bit code units.

I would also just generally like to know why you can't just have in a language "unicode", and there is no utf8/utf16 encoding/decoding, because it is already in let's say utf8 format. This makes me wonder if JavaScript needs to do this encoding/decoding for some reason, such as maybe because JavaScript uses utf16 encoding, and so the stream of bits you pass to utf8.js is a stream of utf16 encoded bits. Or maybe it's not that, and instead the stream of bits you pass to utf8.js is a stream of x (something else) such as decimal encoding or whatever that may mean.

I am just not sure what format/encoding the bytes are coming in as, and also why we need to do the conversion. For example, in here when they mention the "Buffer class is ... raw binary data" (in Node.js), I am not sure what that raw binary data means. Maybe that just means it is UTF16, which is why it would then maybe need to be converted to UTF8, I am not sure. And when this mentions JSON.parse doesn't do any string decoding, I start to wonder if this means that JSON.parse assumes UTF16 encoding, or perhaps it assumes UTF8 encoding, I'm not sure. Searching for the "encoding JSON.parse uses" doesn't reveal anything.

  • 1
    One point that the answers don't really drive home: Encoding is how we change a sequence of code points (which usually, but not always, means a sequence of characters) in to a sequence of bytes. I don't know Javascript, but in many modern languages, the thing called string is a sequence of characters, and before you can save it to a file, or transmit it over the net, it must be encoded as a byte sequence. – Solomon Slow Jul 24 at 21:15
up vote 2 down vote accepted

Code points are often characters, but could be other things, such as control characters (carriage return, etc.), white spaces or accents. Code points can be represented by 21-bit numbers from hex 0 to 1FFFFF.

Unicode started off using 16 bits per code point (hex 0 to FFFF), but rather embarrassingly ran out of code points and had to increase it to 21 bits. So the idea of just using a 16 bit number for a code point had to be scrapped.

We have now ended up with three different encodings.

UTF-32 just pads out the 21 bits to a 32-bit number. It's simple but wastes a lot of space.

UTF-16 is the old 16-bit characters, but with a bodge where certain characters are reserved, and sticking two of the reserved ones together makes a 21-bit character. To make it worse, UTF-16 comes in big-endian and little-endian versions.

UTF-8 was invented as a nice clean way to pack Unicode in an efficient way. It uses between 1 and 4 bytes per code point. By a bit of clever design, the ASCII characters are also valid 1-byte UTF-8 characters.

So you need the encoding to convert the 21-bit Unicode code points into a series of 8-bit bytes in UTF-8 format. The sequence of bytes you get after you have encoded them is the stream.

The term "stream" implies that you can just treat it as a series of bytes, and not worry about what they are until you get to the place where you need to decode them again.

  • Or also a whole group of characters, like the ffi-ligature. So, there is an n-m relationship between characters and codepoints. Well, depending on which of the many definitions of "character" you deign to use, don't forget to say. – Deduplicator Jul 24 at 15:51
  • Note that UTF-32 also comes in big-endian and little-endian versions. UTF-16 is no worse than UTF-32 in that regard. UTF-8 is the only encoding here that does not have to worry about endianness (at least until 4-bit computers see a massive resurgence in popularity...) – 8bittree Jul 25 at 16:53

I am not sure what raw binary data means.

It means nothing here is sure what it means.

If I hold up two fingers am I saying 2, V for victory, or the peace sign?

Knowing the encoding lets you know the meaning behind the symbol. With out knowing the encoding all you know is that I'm holding up two fingers. When you process data without knowing the encoding, the meaning, that's raw binary data.

The vocabulary we use for this is weird. We sound like what it is changes just because we understand it. But I'm holding up the same fingers regardless of how well you understand.

What changes is what you can do with it. If you need to tell people what I'm signaling and they've never heard of a victory sign you might show them a fist held high in celebration to show the same thing. You can't do that if you don't know what a victory sign is. All you can do is say "he's holding up two fingers".

Lets try an exercise. If I hold up 5 fingers on one hand and 2 on another you might assume I mean 7. You could be right. But if you see me count through 0 to 5 on my right hand then hold up 1 on my left only to again count though 0 to 5 on my right and then hold up 2 on my left, well you might assume I'm counting by 5's.

They're the same fingers. But how you think about them changes what they mean.

ASCII is one way to think about bits. UTF-8 is another. Raw binary is a way to think about bits that says "I have no clue, let something else figure out what this means. I'm just going to report what I see".

  • Well, it doesn't matter whether I think ASCII or UTF-8, unless it isn't valid in both. That's the beauty of different schemes sometimes having some overlap not only in representable ideas, but even in their representation. In this specific case, it's intentional: One of the design-criteria for UTF-8 was making it one of the myriad extended-ASCII-charsets in existence, with all the advantages that entails. – Deduplicator Jul 24 at 15:49
  • @Deduplicator encoding in ASCII is always encoding in UTF-8. The reverse isn't always true. And encoding is not just about the number of bytes used to encode. EBCDIC uses the same number of bytes as ASCII but still thinks of the bits in a completely different way. – candied_orange Jul 24 at 17:12

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.