Why Unicode Encoding/Decoding is Necessary in JavaScript

Question

I am wondering why unicode encoding is necessary in JavaScript. I am looking at utf8.js as an example. I am also looking at the utf8 spec, but am not really following the different pieces of data. Also, I don't fully understand when and where you are supposed to encode/decode, and what the "current" format is of some set of bytes in a programming language, so that's making it difficult to follow this.

UTF-8’s encoder’s handler, given a stream and code point, runs these steps:...

I would like to know what the stream is, and what the code point is. I understand that code points are characters, which in utf8 might be composed of multiple 8-bit code units.

I would also just generally like to know why you can't just have in a language "unicode", and there is no utf8/utf16 encoding/decoding, because it is already in let's say utf8 format. This makes me wonder if JavaScript needs to do this encoding/decoding for some reason, such as maybe because JavaScript uses utf16 encoding, and so the stream of bits you pass to utf8.js is a stream of utf16 encoded bits. Or maybe it's not that, and instead the stream of bits you pass to utf8.js is a stream of x (something else) such as decimal encoding or whatever that may mean.

I am just not sure what format/encoding the bytes are coming in as, and also why we need to do the conversion. For example, in here when they mention the "Buffer class is ... raw binary data" (in Node.js), I am not sure what that raw binary data means. Maybe that just means it is UTF16, which is why it would then maybe need to be converted to UTF8, I am not sure. And when this mentions JSON.parse doesn't do any string decoding, I start to wonder if this means that JSON.parse assumes UTF16 encoding, or perhaps it assumes UTF8 encoding, I'm not sure. Searching for the "encoding JSON.parse uses" doesn't reveal anything.

Answer 1

Code points are often characters, but could be other things, such as control characters (carriage return, etc.), white spaces or accents. Code points can be represented by 21-bit numbers from hex 0 to 1FFFFF.

Unicode started off using 16 bits per code point (hex 0 to FFFF), but rather embarrassingly ran out of code points and had to increase it to 21 bits. So the idea of just using a 16 bit number for a code point had to be scrapped.

We have now ended up with three different encodings.

UTF-32 just pads out the 21 bits to a 32-bit number. It's simple but wastes a lot of space.

UTF-16 is the old 16-bit characters, but with a bodge where certain characters are reserved, and sticking two of the reserved ones together makes a 21-bit character. To make it worse, UTF-16 comes in big-endian and little-endian versions.

UTF-8 was invented as a nice clean way to pack Unicode in an efficient way. It uses between 1 and 4 bytes per code point. By a bit of clever design, the ASCII characters are also valid 1-byte UTF-8 characters.

So you need the encoding to convert the 21-bit Unicode code points into a series of 8-bit bytes in UTF-8 format. The sequence of bytes you get after you have encoded them is the stream.

The term "stream" implies that you can just treat it as a series of bytes, and not worry about what they are until you get to the place where you need to decode them again.

Answer 2

I am not sure what raw binary data means.

It means nothing here is sure what it means.

If I hold up two fingers am I saying 2, V for victory, or the peace sign?

Knowing the encoding lets you know the meaning behind the symbol. With out knowing the encoding all you know is that I'm holding up two fingers. When you process data without knowing the encoding, the meaning, that's raw binary data.

The vocabulary we use for this is weird. We sound like what it is changes just because we understand it. But I'm holding up the same fingers regardless of how well you understand.

What changes is what you can do with it. If you need to tell people what I'm signaling and they've never heard of a victory sign you might show them a fist held high in celebration to show the same thing. You can't do that if you don't know what a victory sign is. All you can do is say "he's holding up two fingers".

Lets try an exercise. If I hold up 5 fingers on one hand and 2 on another you might assume I mean 7. You could be right. But if you see me count through 0 to 5 on my right hand then hold up 1 on my left only to again count though 0 to 5 on my right and then hold up 2 on my left, well you might assume I'm counting by 5's.

They're the same fingers. But how you think about them changes what they mean.

ASCII is one way to think about bits. UTF-8 is another. Raw binary is a way to think about bits that says "I have no clue, let something else figure out what this means. I'm just going to report what I see".