For instance if I had only ASCII characters and then switched to Unicode now I have to consider special characters, and bigger strings or chars. What else along these lines should I be considering?

I'm assuming a Unicode string contains special characters as well, but other than that, what am I missing?

  • When you say "switched to Unicode" are you asking about converting your running source code or data to Unicode or both .OR. are you speaking generally about using Unicode for a NEW project? – Elliptical view Jan 19 '14 at 3:22

Depending on the encoding, you might get larger char sizes (8 bits->16 bits), though UTF-8 is able to handle a Unicode character with the same size as a standard ASCII character. Bytes are encoded and decoded in Unicode (while ASCII isn't, as it has a 1-to-1 relation with the byte value), giving a minor performance penalty. Different Unicode encodings exist, such as UTF-8 (8 bits), UTF-16 (16 bits) and UTF-32 (32 bit, I have never seen this one in use though).

You get advantages, more characters. This is needed if you want to internationalize your program to a bunch of different lamguages because they may use different characters than the English alphabet does. The little possible space and performance gain is seldom worth sticking to ASCII in case your language has built in support for Unicode.

Some software might have a trouble with Unicode, but that is far from a problem today.

Unicode is a superset of ASCII, you won't lose characters, only gain. As you have more characters, you will also have to handle more of them. Say you have a method for converting a string to lowercase, you will have to handle many more different characters if you used Unicode instead of ASCII.

  • You omitted the "encode/decode" problem. Unicode is encoded into bytes. Bytes are decoded into Unicode. ASCII is simply bytes. – S.Lott May 3 '11 at 19:54
  • 1
    Those would be 7- or 8-bit bytes, depending :) – Frank Shearar May 3 '11 at 20:12
  • 1
    "but that is far from a problem today"... that depends on who you ask :) – Nathan Osman May 4 '11 at 1:02

I've used UTF32 (UCS4) internally; UCS2 is 16 bit characters and it's not quite enough for Unicode. Was a decade ago, but not now.

I use UTF-8 for all I/O. ASCII is a subset of UTF-8 so you rarely have to deal with the extended characters. Just figure antying above 0x7E is some weird foreign language text and preserve it but ignore it.

The big thing you gain from Unicode is that you can support all foreign languages simultaneously. With other encodings you can support German and Russian and Chinese and Thai, but not on the same page. You can even put Chinese language strings in your source code. Make everything UTF-8 and forget about it.

Well, almost; Hebrew and Arabic go right to left and require extra work sometimes.


Don't forget the "going-back" case. If you're moving data from UTF-8 to ASCII (say you have to create a text file to feed into a legacy system), you lose characters and have to consider the active Code Page again.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.