6

I'm creating a library. I want to use it in multiple projects which may use multi-byte or unicode (std::string or std::wstring). I've adopted the old MS method of conditional compiling:

namespace my_namespace {
#ifdef UNICODE
    typedef std::wstring String;
    typedef std::wstringstream StringStream;
    #define Str(s) L##s
#else
    typedef std::string String;
    typedef std::stringstream StringStream;
    #define Str(s) s
#endif
}

(The Str macro is for string literals. VC++ marks wide strings with L. Example: L"this is a wide string";)

Are there better ways to accomplish this?

  • 4
    Why do you think Unicode requires std::wstring? Standard strings manage bytes, not characters. – user22815 Jul 22 '16 at 20:56
  • 1
    Please see "Is TCHAR still relevant?". – Deduplicator Jul 23 '16 at 11:02
  • @Snowman I should have clarified: this is mostly for interfacing with Windows API. The UNICODE define comes from VC++ and determines which version of an API function to call. Ex: SetWindowTextA(HWND, char *) or SetWindowTextW(HWND, wchar_t *). – Johnny Mopp Jul 27 '16 at 14:27
  • @Deduplicator Maybe my question should have been: When developing Windows programs with UNICODE defined, is there any need to use std::string as it would need to always be converted to wchar_t for Windows function calls? (I know that's a lot different from original question...my apologies). I've been using mostly C# these days. Or maintaining old C++ code that uses char *. – Johnny Mopp Jul 27 '16 at 14:34
  • 1
    If your strings are nearly all used in gluing windows api-calls together, yes you should see UTF-16 as the logical format. If it isn't, you should probably see UTF-8 there instead... – Deduplicator Jul 27 '16 at 17:31
1

The old Microsoft technique

The good old Microsoft technique has served millions of applications, so it is definitively to be considered as a valuable and proven approach.

Three remarks:

  • Microsoft uses this conditional compilation not only the few core elements (TCHAR, TEXT, ...), but also for a lot of other string related functions (see example in the MSDN article) in order for this to work consistently.

  • You have to be careful about the combination of macros with namespaces. For example Str() looks like a normal function, but it is a macro defined globally and not limited to your namespace (and to be used without namespace prefix). I'd suggest to use capitals to make this explicit

  • If you start now a new code base, I'd suggest to adopt Meyer's recommendation to prefer type alias over typedef.

Less redundant variant

As in C++ string/wstring, stringstream/wstringstream, etc... are only char/wchar_t specializations of basic_string<X>/basic_stringstream<X>, I'd define the types to be used based on the underlying character type that you want:

namespace mine {
#ifdef UNICODE
    using Char = wchar_t; 
    #define Str(s) L##s
#else
    using Char = char; 
    #define Str(s) s
#endif
    using String = std::basic_string<Char>;
    using StringStream = std::basic_stringstream<Char>;
    // ...  a lot more but only once
}

Demo

If needed, you could then easily switch to char32_t if you'd wanted to work with full 32 bits unicode across all platforms (currently wchar_t on windows is 16 bits and uses UTF16 encoding, whereas on linuts it's 32 bit and UTF32) as you could using u32string).

Conditional compilation

In theory you could imagine a runtime decision whether to run unicode or not. But to achieve this you'd need to create all objects using an abstract factory. This seems very painful and complex. Not speaking of the code bloat having every string function in double.

Another approach could be to use some templates to define the types at compile time using some clever template. But ultimately you'd need to rely on some macro, that you could define in your build scripts to automate building of all the versions. As in the end you'd rely on them, why not facilitate the approach and using them for what they are supposed to do !

1

For what it's worth, std::wstring does nothing what you'd expect (it's UCS-2, not UTF-16, they are different; the former cannot express characters outside the basic multilingual plane, including Emoji such as U+1F44E THUMBS DOWN SIGN 👎).

For information on Unicode handling in C++11 (and later), see this overview.

TL;DR you probably really want something like ICU or Boost.locale. Standard C++ is just woefully inadequate (and overly complex) with Unicode without using UTF-32 everywhere.

  • Why do you think it is UCS-2 instead of UTF-16? Where doesn't it act as you think it should? Indexing and length are defined just as the encoding says... – Deduplicator Jul 26 '16 at 9:02
  • 1
    std::wstring is really just a dumb container of code units, with functions that manipulate its contents on the code unit level. Whether you treat it as UCS-2 or UTF-16 is up to you; the class will be rather unhelpful either way. – Sebastian Redl Jul 26 '16 at 9:49
0

In theory you can just use the TCHAR data type, use the 't' version of all the string functions and compile with the appropriate defines and everything works....

....but in the real world, you will end up with some API calls that are only available in the wrong format and need to convert strings between encodings, and If you do not know the code page encoding of the mbcs string this will be problematic. (The default assumption is that it is encoded in the OS current code page - but this is a dangerous assumption due to networks!)

Other things that go wrong include code that assumes that wchar strings need the same number of characters as char strings and that the only safe conversion of Unicode to MCBS is to specify the UTF8 code page - Unicode characters that don't exist in the target code page get converted into the 'default' characters, which is a great way to lose your Asian and Arabic characters on a US server.

Finally, remember that Unicode letters can need multiple wchar characters to store a single letter,

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.