How does Python compile some its code in C?

Question

I read that some constructs of Python are more efficient because they are compiled in C.

https://wiki.python.org/moin/PythonSpeed/PerformanceTips

Some of the examples used were map() and filter(). I was wondering how Python is able to do this? It's generally interpreted, so how does some of the code get compiled while another is interpreted - and in a different language? Why not just compile the whole thing?

Answer 1

I read that some constructs of Python are more efficient because they are compiled in C.

That's not true.

First off, there is nothing in the specification of the Python language that requires that certain functions have to be implemented in a certain language. A Python implementor can choose to implement any function and language construct anyway he wants. For example, in Jython, those functions are implemented in Java, not in C. In IronPython, they are implemented in C#. In PyPy, they might be implemented in RPython or just in Python. In Pynie, they are probably implemented in Python.

Seondly, there is nothing in the specification of the C language that says that it must be compiled. There are interpreters for C.

Thirdly, just because it is in C doesn't mean it's fast. There are C compilers out there which produce really terrible code, and there are pretty fast Python implementations.

And fourth, even if the function implemented in C is blazingly fast, that doesn't necessarily translate into faster execution speed of the overall program. Optimizations typically don't work across languages. For example, the optimizer can't inline the call to map into your program, because your program is written in Python but map is written in C. But inlining is pretty much the foundation of all optimizations, because inlining (and loop unrolling) give nice long straight paths of code without branches or calls, which is what optimizers love.

It's generally interpreted,

Actually, all currently existing Python implementations always compile Python code, they never interpret it.

Why not just compile the whole thing?

That's a good question! Writing everything in the same language has many advantages, some of which I outlined above. (Another one is that it's easier to find collaborators who know one language than two.)

If everything is written in the same language, then performance improvements to that language get multiplied throughout the whole system. If everything is written in C, then making the Python 10 times faster isn't going to speed up your program much, because most of the code isn't Python. Sure, the code you wrote is running 10 times faster, but that code mostly consists of calls to C functions which have the same speed as before.

But if everything is written in Python, then making Python faster will have a ripple effect: the primitive types get faster, the datastructures built on top of those primitive types get faster, the algorithms using those datastructures get faster, the modules using those algorithms get faster and so on.

Answer 2

The interpreter itself is written in C (or at least the reference implementation CPython is). That means every language feature is implemented with a tiny C function. All the interpreter does is read the sourcecode to find out which of these tiny functions to call.

This means it is not a problem to also delegate more complex functions to C implementations.

These functions are not compiled with your program. They already exist in compiled form in the Python interpreter which just calls them when it encounters them in your sourcecode.

Answer 3

Saying that they are "compiled in C" is a bit confusing statement. Let me rephrase it: CPython (the most widespread implementation of the Python interpreter, to which you are referring) is a C program, and those are C functions built in the interpreter, not Python code that the interpreter magically chooses to compile to C.

When you feed a .py file to Python, it parses it and converts each function into bytecode, i.e. a compact binary form of the operations that the function does . This is fed to the Python virtual machine, a piece of the interpreter which reads the bytecode and does what it tells it to do.

When you call functions written in Python nothing special happens: the VM will just jump to interpret their bytecode; but when you call a function that is built in inside the interpreter, the interpreter calls the corresponding C function that it has built in (this happens all the time, for example most operations on primitive types are actually performed by code compiled directly inside the interpreter).

Function written in C in CPython are generally faster, both because they are written in a language that is tailored for high performance, compiled to native code (no need for an interpreter, runs directly on the CPU) and has simpler, less dynamic semantics (for example, in Python each access to a member is a complex hashtable lookup, in C it can boil down to a single assembly instruction).

It's generally interpreted, so how does some of the code get compiled while another is interpreted - and in a different language? Why not just compile the whole thing?

You seem to think that the it's the interpreter to decide what code is compiled and what is interpreted; that's a wrong premise. The compiled parts are code that it's written directly in C by the CPython developers, and is compiled inside the interpreter when the interpreter is built (other code written in C or other compiled languages can be called by Python if it's compiled in Python extensions or via ctypes). All the CPython interpreter can do when you run it is just to interpret Python and call ready-made C functions.

Answer 4

It's not "compiled in C" per se: map and filter are built-in functions from the standard library, which is provided by the language implementation (in this case, CPython). Since CPython is written in C, it can easily implement any of its primitive functions in C if needed.