-3

From the UTF-16's wikipedia entry, the second sentence states it's a variable length encoding.

But where is the separator between a 16-bit character and 32-bit encoding? I know a lot of characters can be stored in the 16-bit character, so people can optimize UTF-6 that way, but I'm still curious because there could come a time when we could surpass the 16-bit mark.

And no worries, I know UTF-8 is standard. I'm just curious.

9

It sounds like you're asking for the mechanism that allows UTF-16 to represent 32-bit characters, which is called surrogate pairs. That Wikipedia article pretty thoroughly explains it.

Incidentally, we surpassed 16 bits a long time ago. In fact, the main reason UTF-16 is widely used is that the fixed-length UCS-2 encoding became popular back when people thought 16 bits would be enough, then it wasn't, so UTF-16 is the easy-to-port-to option for everyone who started using UCS-2.

  • Oh wow thank you. Sorry, the tutorials about UTF I was reading didn't cover that. – Arrow Jun 17 '15 at 19:33
4

It's a very similar principle to UTF-8. In UTF-8, you can look at a single byte and it is either a "single byte code point", "first of two bytes", "first of three bytes", "first of four bytes" or "continuation of two, three or four byte code point".

In UTF-16, you can look at a single 16 bit value, and it is either a "single 16-bit word code point", or "first of two halves" or "second of two halves": The codes D800 to DBFF are "first halves", the codes DC00 to DCFF are "second halves", and to get a 32 bit code point you combine the last ten bits of each, giving a 20 bit value, and add 0x10000.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.