For what I understand, given a sequence of bytes without any further information, it's not generally possible to understand which encoding we are talking about. Of course we can guess (e.g. perl's Encode::Guess and similar tools), but sometimes this is just not feasible.

In my case, I've got a byte array which is \xe2\x80\xa1, and I can guess from the context (as a human) that it should correspond either to a à character (\xc3\xa0 in UTF-8) or to a á (\xc3\xa1 in UTF-8). It comes from an xml which declares iso8859-1 in the header, and is produced by a third party tool, which is clearly broken, but I have to deal with it. As you can guess the decoding fails, and no luck with Encode::Guess.

How would you face this kind of problem? I know there's no silver bullet, but is there a tool which outstands the others?

  • If the tool claims to generate iso latin-1 outout, and manifestly doesn't, then there is by definition something wrong with it and must be fixed. What other possibility could there be? – Kilian Foth May 11 '15 at 15:00
  • @KilianFoth, exactly, I totally agree with you. Sometimes however you don't have control on it. Let me fix the question btw... – Dacav May 11 '15 at 15:01
  • Never trust the content-type in the XML file. – user22815 May 11 '15 at 15:03
  • To be honest, I have never in my life received an à that was actually supposed an Ã. It was always a German umlaut encoded in UTF8 read by someone expecting Latin1. So in practice, you're pretty safe assuming that anything with \xc3 in it is actually UTF-8 text. (But don't tell anyone I said that, please.) – Kilian Foth May 11 '15 at 15:08
  • 3
    Check out this SO question and its answers: Correcting the XML encoding. It is about a Python program but the general idea should be applicable to any language. – user22815 May 11 '15 at 15:09

XML encoding can be tricky, because some XML generators might hard-code a generic content type such as ISO-8859-1 even if the document contains e.g. UTF-8. Part of the reason is most text is ASCII, and valid (7-bit) ASCII is also valid as most other encodings. Developers might not understand character encoding or might not care (works with my test data!).

One general approach is to attempt to decode the XML using the provided content-type. As with HTML, this is located near the top of the document and no non-ASCII characters should be between the start of the document and this encoding (except for byte order marks which implicitly provide a content-type).

If this encoding fails, then try with one or more predefined content-types. Good candidates are UTF-8, UTF-16, and ISO-8859-1. Plain old (7-bit) ASCII also decodes fine as UTF-8 and ISO-8859-1 so that is implied here as well.

Please note that depending on the language and XML implementation you may be able to "plow through" errors or not: it is best to have it fail fast on errors so you know to try another encoding.

See Also

| improve this answer | |
  • Aha! That absolute minimum is the one in perlunitut, isn't it? :D – Dacav May 11 '15 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.