1

I'm working with a binary structure, the goal of which is to index the significance of specific bits for any character encoding so that we may trigger events while doing specific checks against the profile.

Each character encoding scheme has an associated system record. This record's leading value will be a C++ unsigned long long binary value and signifies the length, in bits, of encoded characters.

Following the length are three values, each is a bit field of that length.

  • offset_mask - defines the occurrence of non-printable characters within the min,max of print_mask
  • range_mask - defines the occurrence of the most popular 50% of printable characters
  • print_mask - defines the occurrence value of printable characters

The structure of profiles has changed from the op of this question. Most likely I will try to factorize or compress these values in the long-term instead of starting out with ranges after reading more.

I have to write some of the core functionality for these main reasons.

  • It has to fit into a particular event architecture we are using,
  • Better understanding of character encoding. I'm about to need it.
  • Integrating into non-linear design is excluding many libraries without special hooks.

I'm unsure if there is a standard, cross-encoding mechanism for communicating such data already. I'm just starting to look into how chardet might do profiling as suggested by @amon. The Unicode BOM would be easily enough (for my current project) if all encodings were Unicode.

Of course ideally, one would like to support all encodings, but I'm not asking about implementation - only the general case.

How can these profiles be efficiently populated, to produce a set of bitmasks which we can use to match strings with common characters in multiple languages?

If you have any editing suggestions please feel free, I am a lightweight when it comes to localization, which is why I'm trying to reach out to the more experienced. Any caveats you may be able to help with will be appreciated.

  • 5
    What is the question? There is a vague description of “concepts” related to “customized auto-detection of character encoding”, but no real problem description. You cannot really expect to design such auto-detection without a solid understanding of character encodings and their great variation (including the fact that not all encodings use a single byte per character, as the question seems to imply). – Jukka K. Korpela Oct 2 '13 at 5:52
  • Maybe add a single paragraph at the beginning summarising the problem - then we can refer to the rest of the post, if need. – matt freake Oct 2 '13 at 9:10
  • @JukkaK.Korpela Apologies if the wording is poor; I will have to rethink how to phrase this. Certainly we know that character encodings can be longer than 1 byte =0 We're trying to filter the first byte first before looking in the rest though. The question is how can an efficient mapping algorithm be implemented while preserving the sort of event hooks and requirements I need, with a few caveats. Anyway I'll think this over and try to edit it. – Garet Claborn Oct 2 '13 at 9:35
4

Your assumptions are mostly wrong, because you do not account for encodings like UTF-16be.

If you take a look at ASCII (or by virtue at UTF-8); the first 32 characters are control characters. This is pretty common for lots of reasons.

No. It happens that ASCII, Unicode codepoints and UTF-8 all have control characters at the first 32 byte/codepoint positions, but this is to ensure backwards compatibility to ASCII! There are many other encodings that do this as well, usually single-byte encodings.

UTF-16 (and predecessors) and UTF-32 encode codepoints with a minimum of 2 resp. 4 bytes. In UTF-16be, a newline is 00 0A, the letter a is 00 7A – in UTF-16le, this is 7A 00.

Control characters for all covered character sets are in the first N bits, hopefully 4.

Haha, what? Control characters do not occupy certain bits, they are usually full bytes. If you are entertaining the idea that control characters are always in a closed range at the beginning of the encoding table, I'll have to disappoint you.

For example, the delete character U+007F encodes to ASCII and UTF-8 7F. Unicode has control characters in the ranges U+0000U+001F and U+007FU+009F. The latter range encodes to multiple bytes in UTF-8, and can't be represented in ASCII.

The most frequently used characters and phrases in a localization are in a particular range smaller than 8 or 8-N bits, but may be contained neighboring bytes (depending on packing order).

I do not quite understand this statement (grammar…). If you mean that there is a single small subrange of byte sequences that is primarily used in a certain language, then you are partially correct – most writers will stick to a specific script. However, consider the following:

  • Some of these ranges are vast. Take for example chinese characters. Unicode contains thousands of CJK characters.
  • Many graphemes have multiple representations. Depending on Unicode normalization, an East Europan writer might produce text with characters from few codepoint ranges, or he may use combining diacritics.
  • Many non-english documents (esp. on the Web) will likely include some latin characters as well.
  • There are dozens of single-byte encodings that only differ above the ASCII range. These encodings are still widely used, and can make it impossible to detect the actual encoding. Compare the encodings Latin-1 to Latin-16.

{ control: 31, range: [65,122], print: [32,255] }

This is wrong for various reasons, some already mentioned. You should realize that encodings trying to be ASCII-compatible will likely not be compatible with each other above including 0x80 (see above). UTF-8 uses bytes in this range to encode the length of the byte sequence of this code point (actually just a few bits, but that puts the resulting bytes here).


Encoding detection can use the following techniques to find a possible encoding:

  • Standards. For example, any XML document is UTF-8 until proven otherwise.
  • Conventions. Windows-1252 is depressingly common.
  • Byte order marks (BOMs) can identify UTF-16 encodings.
  • Exclusion. Many encodings have illegal byte sequences that cannot occur (this is especially helpful to rule out UTF-8). Even in single-byte encodings, some bytes are unassigned.
  • Unique characteristics. If every 2nd byte is NUL, there is a good chance we are seeing UTF-16le (but we might still be wrong).
  • Statistics using external information. What are the odds that data from France will use a Chinese encoding?

Read the Wikipedia article on charset detection to get started. I recommend using an existing solution for your project, e.g. Mozilla's chardet.

  • thanks for your response. Apologies I have difficulty collecting this concept - its my first major project where I'm unable to convert into a Unicode format using a library. I am trying to learn the lingo better so I may reform the question. When I'm talking about control characters and bit ranges; I'm referring to bitmasking primarily. I should note, this { control: 31, range: [65,122], print: [32,255] } is meant only for ascii. Other records would have different values. Based on what you've mentioned I am looking at BOM and order scheme instead.Thanks for the assist. – Garet Claborn Oct 3 '13 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.