Can we make general statements about the performance of interpreted code vs compiled code?

Question

I'm comparing two technologies in order to reach a recommendation for which one should be used by a company. Technology A's code is interpreted while technology B's code is compiled to machine code. In my comparison I state that tech B in general would have better performance since it doesn't have the additional overhead of the interpretation process. I also state that since a program could be written in many ways it is still possible a program written in tech A could outperform one written in tech B.

When I submitted this report for review, the reviewer stated that I offered no clear reason why in general the overhead of the interpretation process would be large enough that we could conclude that tech B's performance would be better.

So my question is can we ever say anything about the performance of compiled/interpreted technologies? If we can say compiled is generally faster then interpreted, how could I convince the reviewer of my point?

Answer 1

No.

In general, the performance of a language implementation is primarily dependent on the amount of money, resources, manpower, research, engineering, and development spent on it.

And specifically, the performance of a particular program is primarily dependent on the amount of thought put into its algorithms.

There are some very fast interpreters out there, and some compilers that generate very slow code.

For example, one of the reasons Forth is still popular, is because in a lot of cases, an interpreted Forth program is faster than the equivalent compiled C program, while at the same time, the user program written in Forth plus the Forth interpreter written in C is smaller than the user program written in C.

Answer 2

Generalizations and specific scenarios are literally opposites.

You seem to be contradicting yourself. On the one hand, you want to make a general statement about interpreted vs compiled languages. But on the other hand, you want to apply that general statement to a concrete scenario involving Technology A and Technology B.

Once you apply something to a concrete scenario, it's not generalized anymore. So even if you can make the case that interpreted languages are slower in general, you're still not making your point. Your reviewer doesn't care about generalizations. You're doing an analysis of two very specific technologies. That's literally the opposite of generalizing.

Answer 3

As a rule of thumb, an interpreted program is about 2x–10x slower than writing the program in the interpreter's host language, with interpreters for more dynamic languages being slower. This is because the interpreted program has to do all the actual work, but additionally has the interpretation overhead.

Depending on the structure of the interpreter, there can be very significant differences. There are two contradictory schools of interpreter design: One says opcodes should be as small as possible so that they can optimized more easily, the other says opcodes should be as large as possible so that we do more work within the interpreter. When the structure of your program matches the philosophy of the interpreter, the overhead becomes negligible.

E.g. Perl is an interpreted language oriented towards text manipulation. An idiomatic Perl program doing text manipulation will not be much slower than a C program, and may even outperform the C program in some cases (possible because Perl uses a different string representation and has various text- and IO-related optimizations built in). However, doing number crunching in Perl is going to be unbearably slow. An increment ++x is a single assembly instruction, but involves multiple pointer traversals and branches for the Perl interpreter. I recently ported a CPU-bound Perl script to C++, and got a 7x–20x speedup, depending on the compiler optimization level.

Speaking about optimizations is important here, since a polished, optimized interpreter may reasonably outperform a non-optimizing naive compiler. Since creating an optimizing compiler is difficult and requires a lot of effort, it is unlikely that your “technology B” has this level of maturity.

(Note: the Computer Language Benchmarks Game site has a confusing structure, but once you reach the table of timings for one problem you'll notice that performance of various languages is all over the place – often, there isn't a clear performance boundary between compiled and interpreted solutions. The most important part of the site isn't the benchmark results, but the discussions on how difficult meaningful benchmarks are.)

When choosing a tech, performance of a language runtime in itself is completely irrelevant. It is more likely to be important that the tech meet some baseline constraints (our budget is $x, we must be able to deliver before yyyy-mm-dd, we must satisfy various non-functional requirements), and that it has a lower total cost of ownership (factoring in developer productivity, hardware costs, business opportunity costs, risk of bugs and unexpected constraints in the tech, maintenance costs, training and hiring costs, …). E.g. in an industry where time-to-market is the most important factor, the tech with the best developer productivity would be the best fit. For a large organization, maintainability and long-term costs may be more interesting.

Answer 4

You absolutely can say something about the performance of compiled/interpreted technologies. But first, you must define "performance". If you're building a computationally simple embedded system, then "performance" would likely lean towards the memory usage side of things. Whereas a computationally complex system operating on large data sets would find itself defining "performance" in the number of computations per unit time since the memory overhead from JVM or .NET would be negligible.

Once you decide on what "performance" is, then you can say something like "we will have 50 billion objects in memory at any given time and interpreted techA adds an additional 8 bytes to each object for internal management which equates to a 400GB memory overhead in comparison to techB which doesnt add this data"

Answer 5

This is a technical question and you already got many good technical answers, but I'd like to point out a slightly different aspect of your situation: the fact that you can't just base a decision like "technology A or technology B" purely on technical and/or performance reasons.

The technical aspects of something like this is just a small part of the decision between A and B. There are dozens of other factors to keep in mind:

• does it involve any licensing costs? For example: you have to pay (a substantial amount) to use a cluster of SQL Server machines vs. using a cluster of PostgreSQL machines.
• do I have employees who are familiar with that technology (stack) and its ecosystem? If yes, are they available? If no, can I hire some? How much will it cost me? Or do I train the existing ones? How much will that cost me?
• what does the client want? This can be an issue a lot of the time.
• is the technology I recommend ready for production use? Or is it just a current hype that maybe will die out? (e.g. think about Node.js when it first came out)
• does the technology I recommend fit well with the existing architecture or the architecture I had in mind? Or do I have to spend a lot of money by making them work together seamlessly?
• and many many more aspects that depend on your specific situation.

As you can see, there is A TON of stuff to consider when making such a decision.

I know this doesn't specifically answer your question, but I think it brings a more big-picture view over your situation and the specifics of such a decision.

Answer 6

Partial evaluation is a conceptual framework relevant to relate interpreters and compilers.

Can we make general statements about the performance of interpreted code vs compiled code?

Programming languages are specifications (written in some report, such as R5RS or n1570). They are not software, so it does not even make sense to speak of performance. But some programming language may have several implementations, including interpreters and compilers.

Even in traditionally compiled languages (that is, languages whose implementations are often compilers) like C, some parts are often interpreted. For example, the format control string of printf (defined in the C standard) is often "interpreted" (by the C standard library, which has a printf function using variable arguments techniques) but some compilers (including GCC) are able -in limited specific cases- to optimize it and "compile" it into lower-level calls.

And some implementations, even within "interpreters", are using JIT compilation techniques (so generate machine code at runtime). A good exemple is luajit. Other implementations (e.g. Python, Ocaml, Java, Parrot, Lua) are translating the source code into a bytecode which is then interpreted.

SBCL is a Common Lisp "compiler" which dynamically translates every REPL interaction (and calls to eval etc...) into machine code. So you feel it is an interpreter. Most JavaScript implementations in browsers (e.g. v8) use JIT compilation techniques.

In other words, the difference between interpreters and compilers is very fuzzy (actually there is a continuum between both), and practically speaking, most programming language implementations has often both an interpreter and a compiler (at least to byte code) facet.

An implementation can be fast or slow independently of using most "compiler" or "interpreter" like techniques.

Some language traits favor an interpreting approach (and can only be efficiently compiled thru whole program analysis).

For some types of problems, designing the software with some metaprogramming approaches is worthwhile and gives important speed-ups. You could imagine that given some specific input, your program dynamically generates specialized code to process it. This is even possible with C or C++ (either using some JIT library, or generating some C code, compiling it as a plugin which gets dynamically loaded).

See also this related question about Python, and that

Answer 7

For code like A = A + B, that can compile down to one or two machine instructions, each taking a certain number of cycles. No interpreter can do the same thing in that number of cycles for a simple reason.

The interpreter also executes an instruction set of its own (call them byte-codes, p-codes, intermediate language, whatever). Each time it sees a byte-code like ADD, it has to look it up somehow and branch to the code that does the addition.

The next time it sees it, it has to repeat that lookup, unless it has a way to remember the prior lookup. If it does have a way to remember the prior lookup, it is no longer what we call an "interpreter", but rather a just-in-time compiler, or JITter.

On The Other Hand...

For code like callSomeFunction( ... some args ...), how many cycles are spent between entering that code and leaving it? It all depends on what happens inside callSomeFunction. It could be a few, and it could be trillions, even if callSomeFunction is itself compiled. If it's a lot, there is no point in debating the interpretation cost of that line of code - the money is elsewhere.

Remember interpreted languages have a value of their own, such as, no need to compile them. (The "compilation" of surface syntax to byte codes takes trivial time. Take R or MATLAB, for example.)

Also, there is flexibility needed for intelligent levels of programming. In Minsky's Society of Mind, Chapter 6.4 B-Brains, there are A programs that deal with the world, and there are B programs that deal with A programs, and there can be further levels. Programs that write and manage other programs can be more easily done in interpretive systems.

In Lisp, you can write (+ A B) to add A and B, but once it is written you only have the choice to run it or not. You can also write (eval (list '+ 'A 'B)) which constructs the program and then executes it. It could construct something different.

The program's subject is another program. This is easier to write in an interpreted language (though, as Jörg points out, newer versions of Lisp, while they have eval, compile-on-the-fly, so they do not have the speed penalty of interpreting).

Answer 8

Kind of, sort of, it depends but as general rule the compiled - be it through JIT or statically compiled - environment will be faster for many compute-intensive tasks - assuming for simplicity same language.

Part of the reason is that interpreted languages need to have interpreter loop - loop which reads an instruction, selects appropriate action to take and executes it. In best case like interpreting Python or Java bytecode (as old JVM did) it have an overhead of few instructions and played havoc with branch predictor - without the last one you can expect huge penalties due to mispredictions. Even a very dumb JIT should speed this significantly.

That said the interpreted language may cheat. For example Matlab have optimized routines for matrix multiplication and with few changes you can get code running on GPU (disclaimer: I work for nVidia - any opinion expressed here is mine and does not represent view of my employer). That way you can write short and powerful higher level code without worrying about details - someone did take care about it and pour time and resources into optimizing it in low-level language. There is nothing inherited about it and it does not prevent, for example, Matlab to JIT the code but often there is no reason as overhead of calling high-level routine is minimal compared with time spent in low-level ones.

TL;DR - the compiled programs have huge performance benefits over the interpreted ones (for apples-to-apples comparison see PyPy Speed). However the speed of executable is only part of problem and it may not contribute much to overall speed (if time is mostly spent in libraries). Also the implementation matters.

Answer 9

Your assumption is well founded, although it is an assumption.

I am not going to go over the reasons why compiled code should be faster than interpretted code: if you know how computers work, that will be obvious. The difference can be orders of magnitude for certain types of problem. If your reviewer seriously disputes that general case, they don't know what they are talking about.

Where they may have a point though is whether the difference is significant in the type of application you are developing. If it is mostly I/O or mostly calling compiled libraries and it doesn't have a lot of computation, the overhead of the interpretation process may indeed be insignificant.

But the point of my post is this: as an I.T. expert you will be often called to make snap decisions based on a general knowledge of how things should work. Doing a specific test might get you a more accurate answer, but it will cost a lot more and it won't get you there first.

But from time-to-time you do get caught out. It's happened to me. You make a good assumption and then you find you failed to take into account the stupidity of the world.

But I can't explain as well as my favourite Dilbert cartoon of all time. Nothing shows better than this the perils of being a smart-ass.

TL;DR: you should be right, but check the real world just in case.

Answer 10

Unless you use something somewhat exotic, your problem won't be about performance about an interpreted language A and compiled language B.

Because if you/your team know A and not B and so write way better code in A than B you can end up having much better performance in A than B. If you have people experienced in one language and the language/libraries can does the job you need, stick to it.

Here's a link about regex in various languages; you'll see that regex are implemented better in some language even if compiled or not: http://benchmarksgame.alioth.debian.org/u64q/performance.php?test=regexdna

Answer 11

I think it's just not a good idea to talk about performance of two technologies just based on the fact that one is compiled and the other one is interpreted. As stated in other answers, it may depend on the area of application (some languages may be optimised to do some operations very quickly and do other things more slowly) as well as on experience of people that are about to use that technology.

I don't think it's reasonable to expect that you will get a performance boost if you take some excellent interpreted language coders and give them some technology that they are unfamiliar with - maybe in theory the latter MAY result in better performance, but in reality, without necessary skills and experience, you won't use all the optimization opportunities.

From one of the well-known Silicon Valley company employees I've also heard that they prefer the language that is simpler to use as it is more costly and troublesome to pay some skilled developers to maintain a complicated, but highly optimized code than just to buy more rig in order to deal with the less efficient implementation, so that also should be considered while choosing the technology.

Answer 12

Once I had to make a similar sweeping statement to justify a big decision.

First, they may not want to believe a humble engineer, so I found some comparable benchmark tests and quoted them. There are a lot of them about, from people like Microsoft, or renowned universities. And they will say stuff like: Method A is between 3 and 10 times faster than method B, depending on variables X and Y.

Second, you might like to run a benchmark of your own, maybe using a representative chunk of the code in question, or something similar you already have. Run it 1000 times overnight so there really is a measurable difference.

At this point the difference (or lack of it) between A and B should be so clear that you only have to present it. So format the results clearly, with diagrams if possible, stating all assumptions and defining all data used.

Answer 13

I would argue that any dynamic language has an advantage over statically compiled ones: "Runtime optimizations"

That is one of the reasons Java can be faster than C++

Yes, loading a dynamically typed language will always have the cost of translation and be at a disadvantage. But once it is running, the interpreter can profile and enhance frequent code paths with runtime information that static languages will never have

NOTE: Well, Java is an interpreted language, not a dynamic one. But it is a great example of what you can speed up with runtime information

Answer 14

... I also state that since a program could be written in many ways it is still possible a program written in tech A could outperform one written in tech B.

When I submitted this report for review, the reviewer stated that I offered no clear reason why in general the overhead of the interpretation process would be large enough that we could conclude that tech B's performance would be better. ...

This would be my approach:

In general, interpreters are compiled, so every interpreted technology is nothing else than a compiled technology if looked at it on a low level. Therefore, compiled technologies just are more and with more possibilities you can never get worse if you are smart (which in general you are). It depends on how much information there is available at compile time and how much information available only at runtime and how good the compilers and interpreters are, but it should theoretically always be possible to at least equal the performance of any interpreter with a suitable compiler, just because interpreters are fabricated by compilers. That it's possible, doesn't mean it's the case for your techs A and B though.

In practice, just tell the reviewer about all the benchmarks available where compiled and interpreted systems are compared. Then ask him to suggest an interpreter that beats your optimized Assembly coded specific algorithm.

One should maybe add, that any such general statement doesn't help at all when comparing two specific techs A and B. There the choice of A and B matters much, much more, than if they are interpreted or compiled.