3

Given that we do not know the encoding of a string what is the best way to make sure that it is transformed to say ASCII? Also in such situations we are willing to accept potential loss of data.

  • 3
    Well known (but not perfect) character set detector here: www-archive.mozilla.org/projects/intl/chardet.html use this to detect the encoding, and then execute the translation according to the source type. It will occasionally be wrong in detection and you will get your "pottential loss of data" there. – Jimmy Hoffa May 6 '13 at 20:13
2

I would follow these steps:

  1. Detect the original character set. Jimmy Hoffa's suggestion of the Mozilla character set detectors is as good as any. Google becomes your friend here too.
  2. Select top one or two choices for character set.
  3. Convert to ASCII
  4. Convert ASCII string back to assumed character set
  5. Compare original string to old to determine amount of information lost during conversion.

You may want to indicate strings that didn't convert correctly along with how far they were off by. That and the assumed source character set may facilitate future conversions.

  • 3
    Converting to ASCII and back doesn't tell you whether the conversion is successful or not, it only tells you whether the conversion process loses data or not. If you convert from any other 7-bit character set to ASCII and back, you should get the same string, but that doesn't mean that you picked the right character set in the first place. You won't lose any data b/c the operation is reversible, but you might not get a readable ASCII string either. – Caleb May 6 '13 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.