How to cross-reference many character encodings with ASCII OR UTFx?

Question

I'm working with a binary structure, the goal of which is to index the significance of specific bits for any character encoding so that we may trigger events while doing specific checks against the profile.

Each character encoding scheme has an associated system record. This record's leading value will be a C++ unsigned long long binary value and signifies the length, in bits, of encoded characters.

Following the length are three values, each is a bit field of that length.

• offset_mask - defines the occurrence of non-printable characters within the min,max of print_mask
• range_mask - defines the occurrence of the most popular 50% of printable characters
• print_mask - defines the occurrence value of printable characters

The structure of profiles has changed from the op of this question. Most likely I will try to factorize or compress these values in the long-term instead of starting out with ranges after reading more.

I have to write some of the core functionality for these main reasons.

• It has to fit into a particular event architecture we are using,
• Better understanding of character encoding. I'm about to need it.
• Integrating into non-linear design is excluding many libraries without special hooks.

I'm unsure if there is a standard, cross-encoding mechanism for communicating such data already. I'm just starting to look into how chardet might do profiling as suggested by @amon. The Unicode BOM would be easily enough (for my current project) if all encodings were Unicode.

Of course ideally, one would like to support all encodings, but I'm not asking about implementation - only the general case.

How can these profiles be efficiently populated, to produce a set of bitmasks which we can use to match strings with common characters in multiple languages?

If you have any editing suggestions please feel free, I am a lightweight when it comes to localization, which is why I'm trying to reach out to the more experienced. Any caveats you may be able to help with will be appreciated.

Answer 1

Your assumptions are mostly wrong, because you do not account for encodings like UTF-16be.

If you take a look at ASCII (or by virtue at UTF-8); the first 32 characters are control characters. This is pretty common for lots of reasons.

No. It happens that ASCII, Unicode codepoints and UTF-8 all have control characters at the first 32 byte/codepoint positions, but this is to ensure backwards compatibility to ASCII! There are many other encodings that do this as well, usually single-byte encodings.

UTF-16 (and predecessors) and UTF-32 encode codepoints with a minimum of 2 resp. 4 bytes. In UTF-16be, a newline is 00 0A, the letter a is 00 7A – in UTF-16le, this is 7A 00.

Control characters for all covered character sets are in the first N bits, hopefully 4.

Haha, what? Control characters do not occupy certain bits, they are usually full bytes. If you are entertaining the idea that control characters are always in a closed range at the beginning of the encoding table, I'll have to disappoint you.

For example, the delete character U+007F encodes to ASCII and UTF-8 7F. Unicode has control characters in the ranges U+0000–U+001F and U+007F–U+009F. The latter range encodes to multiple bytes in UTF-8, and can't be represented in ASCII.

The most frequently used characters and phrases in a localization are in a particular range smaller than 8 or 8-N bits, but may be contained neighboring bytes (depending on packing order).

I do not quite understand this statement (grammar…). If you mean that there is a single small subrange of byte sequences that is primarily used in a certain language, then you are partially correct – most writers will stick to a specific script. However, consider the following:

• Some of these ranges are vast. Take for example chinese characters. Unicode contains thousands of CJK characters.
• Many graphemes have multiple representations. Depending on Unicode normalization, an East Europan writer might produce text with characters from few codepoint ranges, or he may use combining diacritics.
• Many non-english documents (esp. on the Web) will likely include some latin characters as well.
• There are dozens of single-byte encodings that only differ above the ASCII range. These encodings are still widely used, and can make it impossible to detect the actual encoding. Compare the encodings Latin-1 to Latin-16.

{ control: 31, range: [65,122], print: [32,255] }

This is wrong for various reasons, some already mentioned. You should realize that encodings trying to be ASCII-compatible will likely not be compatible with each other above including 0x80 (see above). UTF-8 uses bytes in this range to encode the length of the byte sequence of this code point (actually just a few bits, but that puts the resulting bytes here).

---

Encoding detection can use the following techniques to find a possible encoding:

• Standards. For example, any XML document is UTF-8 until proven otherwise.
• Conventions. Windows-1252 is depressingly common.
• Byte order marks (BOMs) can identify UTF-16 encodings.
• Exclusion. Many encodings have illegal byte sequences that cannot occur (this is especially helpful to rule out UTF-8). Even in single-byte encodings, some bytes are unassigned.
• Unique characteristics. If every 2nd byte is NUL, there is a good chance we are seeing UTF-16le (but we might still be wrong).
• Statistics using external information. What are the odds that data from France will use a Chinese encoding?

Read the Wikipedia article on charset detection to get started. I recommend using an existing solution for your project, e.g. Mozilla's chardet.