I'm kind of horrified that people actually suggest using UTF-8 inside a std::string and are OK with UTF-16 std::wstring.

Do you actually do this? Do you think that it is OK?

  • 2
    Perhaps you could share your thought process, before this gets closed as flamebait? – Robert Harvey Oct 29 '10 at 15:22

My convention is:

  • std::string = UTF-8. For general use.
  • Utf16String = UTF-16. Used mainly for Windows system calls.
    • std::wstring in Windows
    • std::basic_string<uint16_t> in Linux
  • Utf32String = UTF-32.
    • std::basic_string<uint32_t> in Windows
    • std::wstring in Linux.
  • std::vector<unsigned char> = non-string binary data

You just have to get used to thinking of strings as being an array of code points instead of an array of characters. With this view, all string methods work correctly except for find_first_of/find_last_of/find_first_not_of/find_last_not_of.

  • I don't disagree about thinking of strings as arrays of code points, but in many respects that's not practical in a number of applications (think, for example, a password verification routine that must impose restrictions like "at least 8 characters", or a "display name" verification routine that must impose restrictions like "at most 256 characters"). What we need is to bite the proverbial bullet and agree on one damn encoding, instead of having 739 different ones, each with subtle and highly esoteric differences. – Nik Bougalis Nov 17 '12 at 4:40
  • 1
    @NikBougalis A display routine would then better not count combining characters, unless at the beginning of the string. Yes, we really should agree on one encoding. (My favorite one would be UTF-8, but I'd also happily accept UCS-4, a.k.a. UTF-32. UTF-16 mixes the worst of both and is not really an option.) But that would not solve issues like “what does the user perceive as a single character.” – Christopher Creutzig Dec 30 '13 at 16:36
  • 1
    @ChristopherCreutzig Absolutely. Diacritics, combining marks, non-space space characters and so on. It's unclear what “single character” means when dealing with these kinds of encodings. – Nik Bougalis Dec 30 '13 at 19:51

Well, let's say that many programmers still don't know that UTF-16 is a multibyte encoding (they still think that 16-bit wide char is enough to represent all the Unicode characters, but actually they are stuck with the old UCS-2).

However, there's no real drawback in using a wstring for storing UCS-16 text, but you should know that its length doesn't necessarily represent the number of text symbols that will be printed.

  • Well, if you are ready to throw pretty much all methods out of the window, shouldn't you use vector instead of base_string? Having a string implies, that methods actually work. – Let_Me_Be Oct 29 '10 at 17:27

Even if you store UCS-4 in a string, you still can't guarantee a 1:1 relationship between an item stored in the string and (at least a typical user will view as) a single character when it's displayed or printed (if the string happens to contain any combining character sequences).

Now, it's true that with UCS-4 a single item in the string maps to a single code-point even if that's not necessarily a single character.You can avoid multiple string elements mapping to a single code point. You can't (entirely) avoid multiple code points mapping to a single "character" (and canonicalization to even get close is non-trivial). The problem is that "code point" is an almost entirely artificial construct. Getting excited about/condemning one while ignoring the other doesn't (IMO) make a whole lot of sense.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.