What are the absolute, minimal requirement for containing SE Asian charsets for an embedded system project?

Question

I work for a company that has begun integrating embedded computer systems into our products that we manufacture. We have a pretty wide range of products and they are distributed across the globe. Additionally, we've designed a few integrated boards that can serve multiple purposes depending on the firmware that has been flashed to the system. This way we don't have to redesign our computer hardware for our various products-- all we have to do is re-write the firmware layer to meet the specific product's needs.

Because of these hardware limitations, changing our hardware takes an act of congress but writing new software is much simpler.

One of our products has a new requirement that we haven't had to previously implement, which is the need for user-inputted text.

Currently, we've been able to store international text in resources and only the necessary font characters are compiled to bit-mapped images. This means that we've been able to store highly-ideographic languages like Chinese and Japanese text in a minimal amount of space because we only use a very small percentage of the entire language set.

Since this new product will require that our users input text, we will have to implement an extensive character set. As primarily a PC developer, I'm quite familiar with ASCII, Unicode, UTF-8, etc., however, implementing a full character set of any of these languages isn't feasible because we have a limited amount of FRAM on the board to store the font data.

My management is hoping that there is a minimal character set that can be used for highly ideographic languages. I believe there is a phonetic alphabet for Japanese (the Hiragana?) Are there similar phonetic alphabets for the Chinese, Korean, Vietnamese, etc. languages as well and, if so, could speakers of these languages communicate with such a narrow character set? I'm pretty sure the answer to that question is "absolutely, no" but it's a question worth asking.

Management has set a "soft" requirement that we can only have a limited character set of about 8,000 characters covering all major languages in common use. If this is not possible, we need to look for some form of alternative method for meeting our needs based off of our limited hardware resources.

I'm sure this problem has had to have been solved before. Does anyone have experience working within such constraints while needing an extensive font and character encoding system? If so, what nuggets of wisdom can you offer?

Answer 1

This is an excellent question.

To tackle your answer one language at a time;

Vietnamese

Vietnamese is not using ideographic characters anymore, but its Latin set is quite wide: look at the example to see how many diacritics it uses:

Tiếng Việt, hay Việt ngữ, là ngôn ngữ của người Việt (người Kinh) và là ngôn ngữ chính thức tại Việt Nam. Đây là tiếng mẹ đẻ của khoảng 85% dân cư Việt Nam, cùng với gần ba triệu người Việt hải ngoại, mà phần lớn là người Mỹ gốc Việt. Tiếng Việt còn là ngôn ngữ thứ hai của các dân tộc thiểu số tại Việt Nam.

The reason is that every Vietnames syllable has one of six tone marks that impact pronouncing — in addition to having one non-standard consonant glyph and six non-standard vowels.

Unicode composes tone marks over vowels; if you have an ability to compose glyphs, you'll only need 13 extra glyphs for Vietnamese, but if not, you'll need 1 extra consonant + 12 vowels * 6 tones + 6 new vowels toneless = 79 extra glyphs, in downcase and uppercase.

Korean

Korean is bad news. Korean is written though an alphabet called Hangul, which, while technically being an alphabet of only 68 letters (called jamo), is actually written in syllable-sized blocks, constructed out of jamo.

An example of how Korean text looks:

한글 또는 조선글은 한국어의 고유 문자로서, 1443년 조선 제4대 임금 세종이 훈민정음(訓民正音)이라는 이름으로 창제하여 1446년에 반포하였다. 이후 한문을 고수하는 사대부들에게는 경시되기도 하였으나, 조선 왕실과 일부 양반층과 서민층을 중심으로 이어지다가 1894년 갑오개혁에서 한국의 공식적인 나라 글자가 되었고, 1910년대에 이르러 한글학자인 주시경이 '한글'이라는 이름을 사용하였다.

Unicode has 11,172 completed block characters — but if you're willing to code the logic to compose the final “blocks” yourself, you can save on character set greatly.

Basically, all syllables can be split in two categories — consonant+vowel and consonant+vowel+final, where final can be a vowel, a consonant or a composite. CV syllables are constructed with C on left and V on right; CVF consonants are composed with CV block on top (left-to-right), and final on bottom.

So, basically, you need:

• 19 initials, in two forms
• 21 medials, in two forms
• 28 finals

for a total of 108 symbols. (I'm not absolutely sure there are no “ligatures” in Korean, so that sometimes a constructed block looks different than the combination of components, but that's the best we'll get for now).

Japanese

As you've correctly noticed, Japanese has a phonetic alphabet — but actually, not just one, but two! Hiragana and Katakana are syllabaries, both with 48 of the same syllables, but used in different contexts (katakana is used for foreign words, hiragana is used for grammar).

Sadly (for our purposes), Japanese is almost impossible to fully write using only these two alphabets — Chinese characters, or kanji, as they're known in this context, are essential to any Japanese text.

Obligatory writing example:

仮名（かな）とは、漢字をもとにして日本で作られた文字のこと。現在一般には平仮名と片仮名のことを指す。表音文字の一種であり、基本的に1字が1音節をあらわす音節文字に分類される。漢字に対して和字（わじ）とも言う。ただし和字は和製漢字を意味する事もある。

Besides kanji, you'll need 103 glyphs to map two phonetic alphabets + 7 common kanji that do not exist in Chinese.

CKJ Punctuation

Not an expert on this one, but both Chinese and Japanese use classical, East Asian punctuation. Unicode has 64 symbols dedicated to CJK punctuation and symbols.

Chinese

So, we have 7631 characters remaining in our “budget”. Will it be enough to cover Chinese characters?

漢字為上古時代的華夏族人所發明創製並作改進，目前确切歷史可追溯至約公元前1300年商朝的甲骨文、籀文、金文。再到秦朝的小篆，發展至漢朝隸變，至唐代楷化為今日所用的手寫字體標準——正楷。汉字是迄今为止连续使用时间最长的主要文字，也是上古时期各大文字体系中唯一传承至今的文字，期間東亞諸國都有一定程度地自行創製漢字。

With over 100,000 characters in existence, it is essentially impossible to fully cover Chinese characterset, the actively used subset is much smaller. 2000-3000 characters are said to be enough for general literacy (HSK, TOEFL-like test of Mandarin Chinese, requires knowledge of 2800 characters for its highest level, HSK Advanced), 4000-5000 characters are enough for an educated person.

Keeping in mind that there are Simplified and Traditional characters (former are used in PRC, latter - in Taiwan), which differ for a lot of characters, remaining 7600 symbols, I'd say, would be just enough to cover most use cases for both charactersets.

Feel free to ask if you have any questions!