I read that some constructs of Python are more efficient because they are compiled in C.


Some of the examples used were map() and filter(). I was wondering how Python is able to do this? It's generally interpreted, so how does some of the code get compiled while another is interpreted - and in a different language? Why not just compile the whole thing?

  • When you write map(something...) in Python, nothing gets compiled to C. It calls code that was already written in C. Imagine that there was a piece of code in the Python interpreter like this: if(!strcmp(functionName, "map")) {/* mapping code goes here */} else if(!strcmp(functionName, "filter")) {/* filter code goes here */} else if ... else {error("Unknown function!");} – user253751 Aug 24 '14 at 2:57
  • You may benefit from reading this: programmers.stackexchange.com/questions/24558/… – Pharap Aug 24 '14 at 7:15
  • @immibis so what happens when you do os.getcwd() for example? os.getcwd is not builtin. Does that also correspond to some C code embedded in the CPython interpreter as in the case of map(something)? – multigoodverse Mar 2 '19 at 13:01
  • @multigoodverse os.getcwd is builtin. If you print(os.getcwd) it prints <built-in function getcwd> does it not? – user253751 Mar 3 '19 at 8:06
  • @immibis you are right. os is not builtin, but os.getcwd is. I used a bad example to illustrate my question, sorry. – multigoodverse Mar 3 '19 at 21:11

I read that some constructs of Python are more efficient because they are compiled in C.

That's not true.

First off, there is nothing in the specification of the Python language that requires that certain functions have to be implemented in a certain language. A Python implementor can choose to implement any function and language construct anyway he wants. For example, in Jython, those functions are implemented in Java, not in C. In IronPython, they are implemented in C#. In PyPy, they might be implemented in RPython or just in Python. In Pynie, they are probably implemented in Python.

Seondly, there is nothing in the specification of the C language that says that it must be compiled. There are interpreters for C.

Thirdly, just because it is in C doesn't mean it's fast. There are C compilers out there which produce really terrible code, and there are pretty fast Python implementations.

And fourth, even if the function implemented in C is blazingly fast, that doesn't necessarily translate into faster execution speed of the overall program. Optimizations typically don't work across languages. For example, the optimizer can't inline the call to map into your program, because your program is written in Python but map is written in C. But inlining is pretty much the foundation of all optimizations, because inlining (and loop unrolling) give nice long straight paths of code without branches or calls, which is what optimizers love.

It's generally interpreted,

Actually, all currently existing Python implementations always compile Python code, they never interpret it.

Why not just compile the whole thing?

That's a good question! Writing everything in the same language has many advantages, some of which I outlined above. (Another one is that it's easier to find collaborators who know one language than two.)

If everything is written in the same language, then performance improvements to that language get multiplied throughout the whole system. If everything is written in C, then making the Python 10 times faster isn't going to speed up your program much, because most of the code isn't Python. Sure, the code you wrote is running 10 times faster, but that code mostly consists of calls to C functions which have the same speed as before.

But if everything is written in Python, then making Python faster will have a ripple effect: the primitive types get faster, the datastructures built on top of those primitive types get faster, the algorithms using those datastructures get faster, the modules using those algorithms get faster and so on.

  • This gave me a lot of insight about how languages can work, thank you (and all the other responders as well). I knew programming languages were complicated, but there is so much information about them and so many idiosyncrasies to each one. – Howcan Aug 24 '14 at 0:58
  • 7
    I think "all currently existing Python implementations always compile Python code, they never interpret it" is overstating the case. There is no implementation of the full Python language that compiles entire programs ahead-of-time to machine code, which is what one normally means when one says X is a compiled language. The most commonly used implementations compile ahead-of-time to some sort of byte code, which is then executed by an interpreter. I haven't seen a language that interpreted the source code directly since the days of C64 BASIC. – zwol Aug 24 '14 at 1:22
  • @Zack Heck, even C64 BASIC didn't interpret the source code directly. – fluffy Aug 24 '14 at 5:24
  • 1
    @Zack: "Compile" doesn't mean "translate to machine code", it means "translate to a different language" (or actually the same one). – Jörg W Mittag Aug 24 '14 at 6:48
  • 1
    @Zack: You are vastly overestimating the cleverness of the Ruby community :-D Until very recently, the most widely used Ruby implementation was MRI, which was a pure AST-walking interpreter. For about 10 years, MRI was the only Ruby implementation and for the next 10, it was still the most widely used. The second production-ready Ruby implementation was JRuby, which started out as a pure AST-walking interpreter, although today it supports both JIT compilation to JVM bytecode and AOT compilation to JVM bytecode as well. (However, it still has the interpreter, because sometimes that's faster.) – Jörg W Mittag Aug 24 '14 at 6:51

The interpreter itself is written in C (or at least the reference implementation CPython is). That means every language feature is implemented with a tiny C function. All the interpreter does is read the sourcecode to find out which of these tiny functions to call.

This means it is not a problem to also delegate more complex functions to C implementations.

These functions are not compiled with your program. They already exist in compiled form in the Python interpreter which just calls them when it encounters them in your sourcecode.

  • 1
    "That means every language feature is implemented with a tiny C function." Not quite true, there are some stuff in the stdlib (which I consider to be a language feature) that's written in Python and not C. – Ayrx Aug 24 '14 at 3:14
  • @TerryChia But what does that Python code in the stdlib boil down to in the end? The most basic language elements which are then implemented in C. When you dig down, you will always end up in a C implementation. – Philipp Aug 24 '14 at 10:35
  • No offense at all :) do not think so :) +1 – user167772 Oct 26 '15 at 9:15

Saying that they are "compiled in C" is a bit confusing statement. Let me rephrase it: CPython (the most widespread implementation of the Python interpreter, to which you are referring) is a C program, and those are C functions built in the interpreter, not Python code that the interpreter magically chooses to compile to C.

When you feed a .py file to Python, it parses it and converts each function into bytecode, i.e. a compact binary form of the operations that the function does . This is fed to the Python virtual machine, a piece of the interpreter which reads the bytecode and does what it tells it to do.

When you call functions written in Python nothing special happens: the VM will just jump to interpret their bytecode; but when you call a function that is built in inside the interpreter, the interpreter calls the corresponding C function that it has built in (this happens all the time, for example most operations on primitive types are actually performed by code compiled directly inside the interpreter).

Function written in C in CPython are generally faster, both because they are written in a language that is tailored for high performance, compiled to native code (no need for an interpreter, runs directly on the CPU) and has simpler, less dynamic semantics (for example, in Python each access to a member is a complex hashtable lookup, in C it can boil down to a single assembly instruction).

It's generally interpreted, so how does some of the code get compiled while another is interpreted - and in a different language? Why not just compile the whole thing?

You seem to think that the it's the interpreter to decide what code is compiled and what is interpreted; that's a wrong premise. The compiled parts are code that it's written directly in C by the CPython developers, and is compiled inside the interpreter when the interpreter is built (other code written in C or other compiled languages can be called by Python if it's compiled in Python extensions or via ctypes). All the CPython interpreter can do when you run it is just to interpret Python and call ready-made C functions.


It's not "compiled in C" per se: map and filter are built-in functions from the standard library, which is provided by the language implementation (in this case, CPython). Since CPython is written in C, it can easily implement any of its primitive functions in C if needed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.