1

My source XML does NOT contain the root declaration i.e.

<?xml version="1.0" encoding="utf-8" ?>

And when I run the file command on it, I get:

ASCII text, with very long lines, with CRLF line terminators

Can I confidently use US-ASCII or UTF-8 interchangeably when reading the XML ?

5

US-ASCII is a 7-bit code, and it's a true subset of UTF-8. In other words, every ASCII file is by definition also an UTF-8 file. The file command classifies it as 'ASCII' because there are no 8-bit characters in it, and it's totally right in doing that.

Nevertheless you should always read XML files assuming that they're UTF8-encoded. It doesn't hurt even if there aren't any 8-bit characters in it, but if there are, reading it as ASCII would simply be wrong.

  • 1
    Thanks. In that case, it sounds like people should deprecate US-ASCII. – socgen hacker Nov 24 '14 at 21:56
  • 3
    And people should teach certain parts of the world that there are öthær characters ουτσιδε of US-ASCII that do exist not just for fun, rather they are актуалли used by 人民. SCNR. – JensG Nov 25 '14 at 0:10
3

ASCII is a subset of UTF-8. You can read any ASCII-encoded document as UTF-8, and it will work. ASCII only uses 7 bits, and UTF-8 uses the unused eight bit to mark non-ASCII code units.

The XML spec has an informal algorithm for detecting the encoding which is necessary to read the <?xml declaration, if one is present. An encoding might also be implied by a Byte Order Mark at the beginning of the document.

If no encoding information is present in the document, you may have encoding information from a transport protocol, e.g. HTTP or MIME. Then use that.

But the default assumption is UTF-8. Citing the XML 1.0 and XML 1.1 spec:

In the absence of information provided by an external transport protocol […], it is a fatal error […] for an entity which begins with neither a Byte Order Mark nor an encoding declaration to use an encoding other than UTF-8. Note that since ASCII is a subset of UTF-8, ordinary ASCII entities do not strictly need an encoding declaration.

Something that appears to be encoded in UTF-8 and only uses the ASCII-compatible subset in the beginning might actually be encoded in any other ASCII-compatible encoding, such as an encoding in the ISO 8859 family (e.g. Latin-9). So you still need to determine the proper encoding through encoding declarations, BOMs, or transport metadata. Do not try to sniff the encoding. If the encoding does not match the actual contents, that's an error and you must not process the malformed document.

  • 1
    I do not think item 2. is correct. XML, as of version 1.0, does not require an XML declaration. But of course it is nice to have one. The default encoding is UTF-8. – Jeppe Stig Nielsen Nov 16 '18 at 12:51
  • @JeppeStigNielsen it seems you're right! XML 1.0 does not require the XML declaration but XML 1.1. does (so that the version can be determined unambiguously). I've fixed the answer. – amon Nov 16 '18 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.