# What is really happening when we change encoding in a string?

http://php.net/manual/en/function.mb-convert-encoding.php

Say I do:

$encoded = mb_convert_encoding ($original);

That looks like simple enough. WHat I am imagining is the following

$original has a pointer to the way the string is actually encoded. Something like char * kind of thing. And then there are things like what the character actually encoded. It's probably somewhere along UTF-64 kind of thing where each glyph is indeed a character. Now when we do$encoded = mb_convert_encoding (\$original);

several thing can happen:

• the original internal representation doesn't change however it is REINTERPRETED so that the code that show up differs
• the original string that it represent doesn't change however the ENCODING change.

Which one is right?

• nameless downvote trolls. :( – user4951 Jun 1 '12 at 5:57
• Voting is anonymous, no one is required to explain their down votes. – yannis Jun 1 '12 at 6:03
• I know. But why would anyone downvote this question? Any reason? – user4951 Jun 2 '12 at 14:29

A string isn't a bag of bytes and a specification of the encoding : in most languages a string is a bag of bytes, and in best cases an implicit encoding defined by the language. It's your responsability as a coder to know what to do depending on the encoding (edit : as tdhammer pointed, most modern language explicitly forbids manipulating strings in other encoding than the implicitely defined one).

When you use an encoding function, you don't set a mention somewhere that the string uses a different encoding, you changes the bytes (the internal representation) so that it is correct regarding the intended encoding. But you must keep trace yourself of the used encoding.

For example, if you change the encoding of a string to UTF-16, strlen will give you an incorrect result (at least if you expect it to be the number of characters) because it will count the bytes. You must use mb-strlen and precise the encoding.

• Actually, many languages (the entire .NET ecosystem, Python, Haskell, ...) treat strings as just that, strings, and convert them to bags-of-bytes when required. – tdammers Jun 1 '12 at 10:40
• I think that .Net internally uses UTF-16 just like Java. And you can't legally have in Haskell a string that is in another encoding than UTF-16, but you're right in that Haskell strings are arrays of Char (and not bytes). – Denys Séguret Jun 1 '12 at 10:49
• IIRC, it's UTF-8, but that doesn't matter. What matters is that the internal string encoding is transparent, that is, they could change it to UTF-16 or UTF-32 at will, without breaking anything at the API level. A string is, conceptually, a different data type than a byte array, and going from one to the other requires an explicit conversion. In PHP, by contrast, both are just strings (just like in C), and you can assign them raw, breaking the encoding in horrible ways. – tdammers Jun 1 '12 at 10:53
• Also, UTF-16 is part of Unicode. – tdammers Jun 1 '12 at 10:53
• @tdammers what you say is right, of course, that's what I meant by an implicitly defined encoding for some languages, versus what OP seems to imply : an encoding different for each string and accompanying the string. – Denys Séguret Jun 1 '12 at 11:01

The problem is that PHP treats strings as bags-of-bytes, not making any distinction between bytestrings and 'proper' strings, and it doesn't track the encoding of these bytestrings.

An encoding is, basically, a mapping of byte sequences to proper string codepoints. So what mb_convert_encoding does is split the given string into appropriate byte sequences according to the input encoding, translate them into code points, and then back into byte sequences based on the target encoding.

Since many encodings do not define all the code points, and some encodings may treat certain byte sequences as invalid (e.g. ASCII does not allow anything above 127), you can specify what PHP should do when it encounters such a situation.

Note that mb_convert_encoding has two parameters that specify the input and output encodings (to and from); if you don't specify the from_encoding, the currently set internal encoding is used. You should always set the internal encoding to UTF-8, unless you have convincing reasons not to.

• What do you mean by PhP not keeping track the encoding of these bytestrings? Does PhP use UTF-8? – user4951 Jun 2 '12 at 14:33
• Where can I learn more about it? I am imagining that PhP must have a class "String" which contain a char * and an enum encoding kind of thingy. Am I wrong? – user4951 Jun 2 '12 at 14:34
• @JimThio: PHP stores strings as byte sequences, and its string functions are completely ignorant of any encoding - they just operate on bytes. There is a multibyte-string API, but it requires you to tell it what the encoding is, and strings are still byte sequences. – tdammers Jun 2 '12 at 14:35
• PHP does not have a String class; the string type is wired into the language, and it stores an array of bytes and a length, but no encoding. You can globally set an internal encoding, and you can pass an encoding to some functions, but that's about it. – tdammers Jun 2 '12 at 14:37