In python, what or who is character encoding information for?

Question

If you go to www.htmlbasictutor.ca/character-encoding.htm you will find the following description of character encoding.

Character encoding tells the browser and validator what set of characters to use when converting the bits to characters.

That's all fine and good - but if we were talking about python instead of html, what would we replace browser and validator with? How would you define those replacement words?

Sidenote: Please mention if your answer is correct for other programming languages.

Answer 1

what would we replace browser and validator with?

"The python interpreter"

When you call .encode()/.decode() method on as string, the encoding defines how the bits in the file gets translated into Unicode characters.

Answer 2

Python itself. More specifically string manipulation functions.

Most string manipulation functions are expected to work on characters, not bytes, so they need to know which bytes are to be considered as "one character" to work.

This is of course a problem in any programming language. The only alternative is for the programming language to assume a specific encoding, forcing any developer that needs or prefers to use a different one to treat the data as binary.

This is the case for older languages, made when ASCII-127 was a reasonable choice and all encodings were single-byte anyway. Most of them fixed this issue later. As an example, in PHP the historical string functions assume one-byte-per-character strings. Later, all those functions have been rewritten to be encoding-aware. The new functions were added to the standard library with an mb_ (multi-byte) prefix to mantain back-compatibility.