Given that we do not know the encoding of a string what is the best way to make sure that it is transformed to say ASCII? Also in such situations we are willing to accept potential loss of data.
I would follow these steps:
- Detect the original character set. Jimmy Hoffa's suggestion of the Mozilla character set detectors is as good as any. Google becomes your friend here too.
- Select top one or two choices for character set.
- Convert to ASCII
- Convert ASCII string back to assumed character set
- Compare original string to old to determine amount of information lost during conversion.
You may want to indicate strings that didn't convert correctly along with how far they were off by. That and the assumed source character set may facilitate future conversions.