9

I'm working on an cross platform C++ project, which doesn't consider unicode, and need change to support unicode.

There is following two choices, and I need to decide which one to choose.

  • Using UTF-8 (std::string) which will make it easy to support posix system.
  • Using UTF-32 (std::wstring) which will make it easy to call windows API.

So for item #1 UTF8, the benefit is code change will not too many. But the concern is some basic rule will broken for UTF8, for example,

  • string.size() will not equal the character length.
  • search an '/' in path will be hard to implement (I'm not 100% sure).

So any more experience? And which one I should choose?

3
  • 7
    wstring is a basic_string<wchar_t>. wchar_t is implementation-defined in size and definitely not enough bits for UTF-32 on Windows. – Lars Viklund Apr 17 '14 at 13:05
  • 2
    Actually, searching for a "/" in UTF-8 is trivial. The guys who designed UTF-8 actually used their brains. – gnasher729 Jun 30 '15 at 16:42
  • 1
    win32 API is not UTF-32, it's UTF-16. The width of wchar_t varies between platforms. – Peter Green May 9 '16 at 11:11
26

Use UTF-8. string.size() won't equal the amount of code points, but that is mostly a useless metric anyway. In almost all cases, you should either worry about the number of user-perceived characters/glyphs (and for that, UTF-32 fails just as badly), or about the number of bytes of storage used (for this, UTF-32 is offers no advantage and uses more bytes to boot).

Searching for an ASCII character, such as /, will actually be easier than with other encodings, because you can simply use any byte/ASCII based search routine (even old C strstr if you have 0 terminators). UTF-8 is designed such that all ASCII characters use the same byte representation in UTF-8, and no non-ASCII character shares any byte with any ASCII character.

The Windows API uses UTF-16, and UTF-16 doesn't offer string.size() == code_point_count either. It also shares all downsides of UTF-32, more or less. Furthermore, making the application handle Unicode probably won't be as simple as making all strings UTF-{8,16,32}; good Unicode support can require some tricky logic like normalizing text, handling silly code points well (this can become a security issue for some applications), making string manipulations such as slicing and iteration work with glyphs or code points instead of bytes, etc.

There are more reasons to use UTF-8 (and reasons not to use UTF-{16,32}) than I can reasonably describe here. Please refer to the UTF-8 manifesto if you need more convincing.

3
  • 2
    I am not really convinced. AFAIK - UTF32 encodes ALL code points into 4 bytes. As such utf32string.size() == number of characters (Please prove me wrong!). As such - from a coding and simplicity point of view, UTF32 is the ONLY encoding avoiding multi word handling complexity and as such allows for less complex and less error prone coding. Only a pity, that C++17 deprecates a lot of utf32 related stuff in ``<codecvt>, <locale>, ..." headers. And Win32 API does not have a MultiByteToUTF32() kind of function (only utf8->utf16). – BitTickler Aug 10 '19 at 0:02
  • @BitTickler While it's true that each codepoint can fit into a single char32_t, not all Unicode characters fit into a single codepoint. For instance, if you want to type an emoticon with a different skin tone, you need two codepoints instead of just one. So the following string of emojis is three characters encoded as 5 codepoints: 👋👋🏻👋🏿 coded as (U+1F44B U+1F44B U+1F3FB U+1F44B U+1F3FF). Some glyphs can be represented in multiple ways. For instance, you can write ñ as two codepoints (U+006E U+0303) or as one codepoint (U+00F1). – Tom Norton Feb 9 at 16:59
  • @TomNorton Yes - as soon as people set out to make a 100% solution it turns ugly. And turns otherwise small projects into huge projects. (Imagine the author of the question starts to use ICU-TS libraries and the complexity of doing it right 100% increases the effort for his project by a factor 10...). So, in order to keep things sane, I am a proponent of "doing it well enough". In that spirit, it might be a good choice to say "I support UTF32 single code point only". And might get away with just a using string32 = std::basic_string<char32_t> and a few helper functions. – BitTickler Feb 10 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.