I used to think that the BOM is optional for UTF-8, but mandatory for UTF-16 and UTF-32.

But then I have read the following (in this article):

Let's look just at the ones that Notepad supports.

8-bit ANSI (of which 7-bit ASCII is a subset). These have no BOM; they just dive right in with bytes of text. They are also probably the most common type of text file.

UTF-8. These usually begin with a BOM but not always.

Unicode big-endian (UTF-16BE). These usually begin with a BOM but not always.

Unicode little-endian (UTF-16LE). These usually begin with a BOM but not always.

So is Notepad not complying with the Unicode standard, or does the Unicode standard says that the BOM is optional for UTF-16?

  • 2
    The manner in which the Byte Order Mark is used is described in detail in RFC 2781, Sections 3.2 and 3.3. For the most part, the language used is SHOULD and MAY, except where it says MUST NOT. The only time the BOM seems to be mandated is in some specific data formats that require it. – Robert Harvey Apr 28 at 5:22
  • 1
    You really shouldn't write BOMs for UTF8 documents; they interfere with one of the biggest benefits of UTF8 - stuff that expects ANSI-encoded text can often work with it without doing any harm. An implementation of cat that thinks of strings as ANSI text will concatenate non-BOM UTF8 text fine, but will leave a spurious BOM if you concatenate two UTF8 files with a BOM. – James Picone Apr 30 at 7:53

The BOM is entirely optional. However, in order to decode UTF-16 you need to know the correct byte order. If you decode with the wrong byte order, you will generally also get valid codepoints. To know the correct byte order, we

  • either know the byte order from an external source, e.g. documentation that states “this tool will always produce UTF-16LE output”,
  • or the encoded text contains a BOM.

For example, the XML standard is defined in a way so that XML documents can optionally start with a BOM, but the <?xml declaration at the start can also be used to determine the encoding.

Editors or web browsers have to work reasonably even when BOMs are missing, and the encoding is ambiguous. They can use statistical data e.g. on expected character frequencies to take a guess, but in the end the user should be able to override the encoding.


If we look at the Unicode Specification (version 10.0), then section 2.6 Encoding Schemes states:

When a higher-level protocol supplies mechanisms for handling the endianness of integral data types, it is not necessary to use Unicode encoding schemes or the byte order mark. In those cases Unicode text is simply a sequence of integral data types.

I.e. as explained above, the BOM is not necessary when we have external information about the byte order. However, there is a requirement that a BOM must be understood by some software dealing with Unicode. From section 23.8 Specials:

Where the byte order is explicitly specified, such as in UTF-16BE or UTF-16LE, then all U+FEFF characters—even at the very beginning of the text—are to be interpreted as zero width no-break spaces. Similarly, where Unicode text has known byte order, initial U+FEFF characters are not required, but for backward compatibility are to be interpreted as zero width no-break spaces. […]

Systems that use the byte order mark must recognize when an initial U+FEFF signals the byte order. In those cases, it is not part of the textual content and should be removed before processing, because otherwise it may be mistaken for a legitimate zero width no-break space.

In section 3.10 Encoding Schemes, various UTF encodings are defined. Here, UTF-16LE, UTF16BE, and UTF-16 are different encodings. The LE and BE variants do not have a BOM. For UTF-16:

The UTF-16 encoding scheme may or may not begin with a BOM. However, when there is no BOM, and in the absence of a higher-level protocol, the byte order of the UTF-16 encoding scheme is big-endian.

Equivalently for UTF-32LE, UTF-32BE, and UTF-32.

So the Unicode Standard does state that the BOM is optional, and mandates how software must handle the presence or absence of a BOM under various circumstances. Unicode-aware software that does not handle encoded text without BOMs is broken.

  • 1
    There is another case where the BOM is unnecessary: if the first character of a file is known to be drawn from a limited subset of all possible characters. For example, JSON is defined such that it may use UTF16 without a BOM because the first character of a valid JSON object is always an ASCII character, so if UTF-16BE is in use, the first byte must be zero, while if UTF-16LE is in use the second byte must be zero (and NULs aren't permitted, so if UTF-8 is in use, neither of the first two bytes may be zero). – Jules Apr 28 at 14:57
  • And if UTF-32 is used, either bytes 0/1 or bytes 2/3 must be zero, and again such JSON could be neither UTF-8 nor UTF-16 since nul codepoints are not allowed in JSON. So you need neither BOMs nor any knowledge which Unicode encoding is used. – gnasher729 Apr 28 at 22:55

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.