Hot answers tagged

813

By default, older versions of IE (<=8) will submit form data in Latin-1 encoding if possible. By including a character that can't be expressed in Latin-1, IE is forced to use UTF-8 encoding for its form submissions, which simplifies various backend processes, for example database persistence. If the parameter was instead utf8=true then this wouldn't ...


131

Unicode is certainly difficult, and the UTF-8 encoding has a couple of inconvenient properties. However, UTF-8 has become the de-facto standard encoding on the web, surpassing ASCII, Latin-1, UCS-2 and UTF-16. Just use UTF-8 everywhere. The most important reason why you should support Unicode is that you shouldn't make unnecessary assumptions about user ...


110

The Unicode standard has lots of space to spare. The Unicode codepoints are organized in “planes” and “blocks”. Of 17 total planes, there are 11 currently unassigned. Each plane holds 65,536 characters, so there's realistically half a million codepoints to spare for an alien language (unless we fill all of that up with more emoji before first contact). As of ...


103

Files generally indicate their encoding with a file header. There are many examples here. However, even reading the header you can never be sure what encoding a file is really using. For example, a file with the first three bytes 0xEF,0xBB,0xBF is probably a UTF-8 encoded file. However, it might be an ISO-8859-1 file which happens to start with the ...


62

I think beyond the technical question, your boss may not have the time to keep up to date on current standards. Since his stance is not completely out to lunch, just out-dated, respect his position when discussing this matter (and you need to remember to discuss, not argue), and try to work through concerns he has with regards to UTF-8. I suspect the ...


49

Which of us is right? Once upon a time, your boss was. But as time goes by, things change. Nowadays, you are (but before running to your boss, be sure to read Nelson's answer too). Old versions of MySQL, and old versions of mostly everything, dealt much better with the older Latin1/ISO-8859-1(5) than UTF8. There is a reason why UTF8 has been created, ...


30

If UTF-8 is actually to be extended, we should look at the absolute maximum it could represent. UTF-8 is structured like this: Char. number range | UTF-8 octet sequence (hexadecimal) | (binary) --------------------+--------------------------------------------- 0000 0000-0000 007F | 0xxxxxxx 0000 0080-0000 07FF | 110xxxxx 10xxxxxx ...


26

This is done so that you can detect when you are in the middle of a multi-byte sequence. When looking at UTF-8 data, you know that if you see 10xxxxxx, that you are in the middle of a multibyte character, and should back up in the stream until you see either 0xxxxxx or 11xxxxxx. Using your scheme, bytes 2 or 3 could easily end up with patters like either ...


23

The choice is not between ASCII and UTF-8. ASCII is a 7-bit encoding, and UTF-8 supersedes it - any valid ASCII text is also valid UTF-8. The problems arise when you use non-ASCII characters; for these you have to pick between UTF-8, UTF-16, UTF-32, and various 8-bit encodings (ISO-xxxx, etc.). The best solution is to stick with a strict ASCII charset, that ...


18

You cannot. If you could do that, there would not be so many web sites or text files with “random gibberish” out there. That's why the encoding is usually sent along with the payload as meta data. In case it's not, all you can do is a “smart guess” but the result is often ambiguous since the same byte sequence might be valid in several encodings.


12

Why do you have this understanding? Both encodings [UTF-8 and UTF-16] can encode all unicode characters by the definition of them being unicode encodings. Anyway, UTF-8 is more optimal for storage and transmission than UTF-16 in your case. Majority of your characters in the files will not be in Chinese but in markup/js syntax. UTF-8 uses 1 byte for those ...


10

Answer to Question 1: why is it that the first 2 bits of every continuation byte are set to “10”? It lets you land at a random place in the sequence and unambiguously work back to the beginning of the current code point (or forward to the start of the next one). If you are starting from the beginning of a sequence, then you know the first byte is a ...


9

It sounds like you're asking for the mechanism that allows UTF-16 to represent 32-bit characters, which is called surrogate pairs. That Wikipedia article pretty thoroughly explains it. Incidentally, we surpassed 16 bits a long time ago. In fact, the main reason UTF-16 is widely used is that the fixed-length UCS-2 encoding became popular back when people ...


8

UTF-8 was specifically designed to be forwards- and backwards-compatible with ASCII, specifically it has these two properties: the encoding of characters within the ASCII character set is the same in UTF-8 as it is in ASCII all other codepoints are encoded as a sequence of 2-6 octets, all of which have their high-order bit (8th bit) set; since ASCII only ...


7

RFC3629 restricts UTF-8 to a maximum of four bytes per character, with a maximum value of 0x10FFFF, allowing a maximum of 1,112,064 code points. Obviously this restriction could be removed and the standard extended, but this would prove a breaking change for existing code that works to that limit. From a data-file point of view, this wouldn't be a breaking ...


6

Most IDEs will default to saving with UTF-8 encoding, and you should almost certainly choose UTF-8 over ASCII when given the option. This is will ensure you don't run into weird problems with internationalization code.


6

For an external encoding (i.e., an encoding of things not inside your program) it is very hard to beat UTF-8; it supports every character your users might ever reasonably need and there's lots of support in many OSes and tools. (The one place that counts as an exception to this is in file names, where you must use the platform's conventions if you want any ...


6

The official way lets the decoder know when it's in the middle of the tuple and it knows to skip bytes (or go backwards) until the byte starts with 0 or 11; this prevents garbage values when a single byte gets corrupted.


6

I am not sure what raw binary data means. It means nothing here is sure what it means. If I hold up two fingers am I saying 2, V for victory, or the peace sign? Knowing the encoding lets you know the meaning behind the symbol. With out knowing the encoding all you know is that I'm holding up two fingers. When you process data without knowing the ...


5

Guessing the encoding of a message based on its byte values is always imperfect, although it can be made to give relatively accurate results. There are libraries available to do this, e.g. IBM's ICU. But the preferable option is always either to standardize client and server on one encoding and always use that, or to force the client to declare the ...


4

If I set my database tables to UTF-8, and set all of my web pages to assume content is UTF-8, is this enough? You need to ensure that the connection between the web application and the database doesn't mangle the encoding (I believe you need to explicitly set this on the connection string for MySQL, for instance). Basically you need to ensure that every ...


4

If it doesn't appear in your payload, any single-byte ASCII character is a valid separator, because the (ASCII) code points 0 - 127 will be unique, no escaped single bytes will match their values. See Wikipedia on UTF-8. Single Byte (ASCII) code points will always be encoded as 0xxxxxxx bits, whereas all bytes of sequences will be encoded as 1xxxxxxx bits. ...


4

Really, only 2 Unicode code-points code stand for infinitely many glyphs, if they were combining characters. Compare, for example, the two ways that Unicode encodes for the Korean Hangul alphabet: Hangul Syllables and Hangul Jamo. The character 웃 in Hangul Syllabels is the single code-point C6C3 whereas in Hangul Jamo it is the three code-points 110B (ㅇ) ...


4

Some situations where restricting the character set only to ASCII may make sense is for limited choice fields, e.g. status fields, because you strictly control the values that can be there, and foreign key/references to external system, because there are rarely any reasons for them to have anything but alphanumeric characters and a few symbols. For any ...


4

It's a very similar principle to UTF-8. In UTF-8, you can look at a single byte and it is either a "single byte code point", "first of two bytes", "first of three bytes", "first of four bytes" or "continuation of two, three or four byte code point". In UTF-16, you can look at a single 16 bit value, and it is either a "single 16-bit word code point", or "...


3

To begin with the answer, it doesn't matter, how your server is configured. The character encoding in MySQL could be configured per-column (means, same table could hold characters in multiple encodings, easy). I.e. my server (and a number of legacy databases in it) is configured for cp1251 by default for old clients that unable to set correct collation upon ...


3

Short answer, your proposal does not differentiate between the first byte and continuation bytes. The bit pattern at the high end of the first byte tells you with how many bytes the actual character is built. These patterns provide also some error recognition while parsing a string. If you are reading the (seemingly) first byte of a character and you get ...


3

Windows Codepages and Extended ASCII support are virtually guaranteed to throw you a curveball.


3

Code points are often characters, but could be other things, such as control characters (carriage return, etc.), white spaces or accents. Code points can be represented by 21-bit numbers from hex 0 to 1FFFFF. Unicode started off using 16 bits per code point (hex 0 to FFFF), but rather embarrassingly ran out of code points and had to increase it to 21 bits. ...


3

I share MrSMith42's experience that anytihn outside of the most simple email has a hard time getting used in many systems that currently exist, however I disagree with their assesment. As it stands the RFC is not widely used, but as the internet keeps booming and non English markets keep gaining in force in the world rather than simply call the RFC "dead ...


Only top voted, non community-wiki answers of a minimum length are eligible