60

I'm comparing two technologies in order to reach a recommendation for which one should be used by a company. Technology A's code is interpreted while technology B's code is compiled to machine code. In my comparison I state that tech B in general would have better performance since it doesn't have the additional overhead of the interpretation process. I also state that since a program could be written in many ways it is still possible a program written in tech A could outperform one written in tech B.

When I submitted this report for review, the reviewer stated that I offered no clear reason why in general the overhead of the interpretation process would be large enough that we could conclude that tech B's performance would be better.

So my question is can we ever say anything about the performance of compiled/interpreted technologies? If we can say compiled is generally faster then interpreted, how could I convince the reviewer of my point?

  • 76
    Since this seems to be a practical problem and not an academic exercise, it might be advisable to not even attempt giving a general answer but actually evaluate technologies A and B. While a general answer probably cannot be given, it is certainly possible to evaluate the performance characteristics of two specific technologies, pointing out their relevance for the kind of applications your organization is interested in. – 5gon12eder Jan 10 '17 at 13:31
  • 26
    The question asks about generalities, but it seems your real problem is with the specifics of Technology A and B. While I think in general interpreted languages may be slower as a whole, that fact likely has absolutely no bearing on your specific technologies. It's quite possible that your specific interpreted B is faster than your compiled A regardless of the general trend. – Bryan Oakley Jan 10 '17 at 13:33
  • 18
    Randomly pick an interpreted language and a compiled one, and bet a dollar that the compiled one is faster. Repeat enough times and you'll eventually come out far enough ahead to buy lunch at Subway. However, there are faster and more interesting ways for a competent programmer to make that much money. And Subway's not all that great anyhow. – Ed Plunkett Jan 10 '17 at 19:03
  • 23
    You seem to be contradicting yourself. On the one hand, you want to make a general statement about interpreted vs compiled languages. But on the other hand, you want to apply that general statement to a concrete scenario. But once you apply it to a concrete scenario, it's not generalized anymore. So even if you can make the case that interpreted languages are slower in general, your reviewer doesn't care about that. You're doing an analysis of two very specific technologies. That's literally the opposite of generalizing. – loneboat Jan 10 '17 at 19:42
  • 8
    Honestly, why the heck do you want to give a company-relevant recommendation for one of two technologies based first and foremost on performance? In special cases (like 3D high speed games programming), performance can be a relevant criterion, but for most real-world business programs, it should be the last thing you compare, not the first. – Doc Brown Jan 10 '17 at 22:33

14 Answers 14

110

No.

In general, the performance of a language implementation is primarily dependent on the amount of money, resources, manpower, research, engineering, and development spent on it.

And specifically, the performance of a particular program is primarily dependent on the amount of thought put into its algorithms.

There are some very fast interpreters out there, and some compilers that generate very slow code.

For example, one of the reasons Forth is still popular, is because in a lot of cases, an interpreted Forth program is faster than the equivalent compiled C program, while at the same time, the user program written in Forth plus the Forth interpreter written in C is smaller than the user program written in C.

  • 12
    as far as I can tell your own prior answer in a linked (possibly a duplicate) question covers these matters much better than this one – gnat Jan 10 '17 at 13:14
  • 11
    If it's impossible to make general conclusions about performance and we can write a certain program in A that outpeforms a similar program in B, but we can also write a program in B that outperforms one written in A. We can not really conclude anything about the speed of language ever then? Anyway Forth looks interesting, I'll have to read more about it later. – EpicSam Jan 10 '17 at 13:35
  • 36
    @EpicSam: Correct. The very idea of "the speed of a language" is fundamentally non-sensical. – Jörg W Mittag Jan 10 '17 at 14:10
  • 23
    In general: be specific. – igouy Jan 10 '17 at 19:57
  • 19
    I assume by "...and some very slow compilers" you meant the code they generate, not their speed of compilation. – martineau Jan 10 '17 at 20:47
81

Generalizations and specific scenarios are literally opposites.

You seem to be contradicting yourself. On the one hand, you want to make a general statement about interpreted vs compiled languages. But on the other hand, you want to apply that general statement to a concrete scenario involving Technology A and Technology B.

Once you apply something to a concrete scenario, it's not generalized anymore. So even if you can make the case that interpreted languages are slower in general, you're still not making your point. Your reviewer doesn't care about generalizations. You're doing an analysis of two very specific technologies. That's literally the opposite of generalizing.

  • 3
    This is by far the best answer to the question text, as opposed to the question title. – David Moles Jan 10 '17 at 22:14
37

As a rule of thumb, an interpreted program is about 2x–10x slower than writing the program in the interpreter's host language, with interpreters for more dynamic languages being slower. This is because the interpreted program has to do all the actual work, but additionally has the interpretation overhead.

Depending on the structure of the interpreter, there can be very significant differences. There are two contradictory schools of interpreter design: One says opcodes should be as small as possible so that they can optimized more easily, the other says opcodes should be as large as possible so that we do more work within the interpreter. When the structure of your program matches the philosophy of the interpreter, the overhead becomes negligible.

E.g. Perl is an interpreted language oriented towards text manipulation. An idiomatic Perl program doing text manipulation will not be much slower than a C program, and may even outperform the C program in some cases (possible because Perl uses a different string representation and has various text- and IO-related optimizations built in). However, doing number crunching in Perl is going to be unbearably slow. An increment ++x is a single assembly instruction, but involves multiple pointer traversals and branches for the Perl interpreter. I recently ported a CPU-bound Perl script to C++, and got a 7x–20x speedup, depending on the compiler optimization level.

Speaking about optimizations is important here, since a polished, optimized interpreter may reasonably outperform a non-optimizing naive compiler. Since creating an optimizing compiler is difficult and requires a lot of effort, it is unlikely that your “technology B” has this level of maturity.

(Note: the Computer Language Benchmarks Game site has a confusing structure, but once you reach the table of timings for one problem you'll notice that performance of various languages is all over the place – often, there isn't a clear performance boundary between compiled and interpreted solutions. The most important part of the site isn't the benchmark results, but the discussions on how difficult meaningful benchmarks are.)

When choosing a tech, performance of a language runtime in itself is completely irrelevant. It is more likely to be important that the tech meet some baseline constraints (our budget is $x, we must be able to deliver before yyyy-mm-dd, we must satisfy various non-functional requirements), and that it has a lower total cost of ownership (factoring in developer productivity, hardware costs, business opportunity costs, risk of bugs and unexpected constraints in the tech, maintenance costs, training and hiring costs, …). E.g. in an industry where time-to-market is the most important factor, the tech with the best developer productivity would be the best fit. For a large organization, maintainability and long-term costs may be more interesting.

  • 10
    If you think "the Computer Language Benchmarks Game site has a confusing structure" then give a URL to the specific page that supports your point, rather than expecting people to search from the top-level looking for something they might never find ! Show; don't just tell. – igouy Jan 10 '17 at 20:01
  • 8
    If you think "The most important part of the site isn't the benchmark results, but the discussions on how difficult meaningful benchmarks are" then give URLs to those pages. Show; don't just tell. – igouy Jan 10 '17 at 20:03
  • 4
    It would be for the convenience of those who read your answer. (I'm just someone who admins the benchmarks game). – igouy Jan 10 '17 at 21:59
  • 2
    @amon linking the k-nucleotide example in the answer would make it more readable and the note more useful, as would linking the discussions (without reading the entire site, it's not obvious to me what discussions you're talking about). – David Moles Jan 10 '17 at 22:12
  • 1
    Where on earth did you get 2-10X from? The ratio depends entirely on how long it takes to fetch and dispatch to the handler for each bytecode as against how long each handler takes to execute, and 10x would only be possible if the fetch cycle took 9x as long as the handler, which is wildly improbably. More usually the execution is dominated by the handler. and the fetch cycle can be vanishingly insignificant. – user207421 Jan 11 '17 at 20:21
18

You absolutely can say something about the performance of compiled/interpreted technologies. But first, you must define "performance". If you're building a computationally simple embedded system, then "performance" would likely lean towards the memory usage side of things. Whereas a computationally complex system operating on large data sets would find itself defining "performance" in the number of computations per unit time since the memory overhead from JVM or .NET would be negligible.

Once you decide on what "performance" is, then you can say something like "we will have 50 billion objects in memory at any given time and interpreted techA adds an additional 8 bytes to each object for internal management which equates to a 400GB memory overhead in comparison to techB which doesnt add this data"

12

This is a technical question and you already got many good technical answers, but I'd like to point out a slightly different aspect of your situation: the fact that you can't just base a decision like "technology A or technology B" purely on technical and/or performance reasons.

The technical aspects of something like this is just a small part of the decision between A and B. There are dozens of other factors to keep in mind:

  • does it involve any licensing costs? For example: you have to pay (a substantial amount) to use a cluster of SQL Server machines vs. using a cluster of PostgreSQL machines.
  • do I have employees who are familiar with that technology (stack) and its ecosystem? If yes, are they available? If no, can I hire some? How much will it cost me? Or do I train the existing ones? How much will that cost me?
  • what does the client want? This can be an issue a lot of the time.
  • is the technology I recommend ready for production use? Or is it just a current hype that maybe will die out? (e.g. think about Node.js when it first came out)
  • does the technology I recommend fit well with the existing architecture or the architecture I had in mind? Or do I have to spend a lot of money by making them work together seamlessly?
  • and many many more aspects that depend on your specific situation.

As you can see, there is A TON of stuff to consider when making such a decision.

I know this doesn't specifically answer your question, but I think it brings a more big-picture view over your situation and the specifics of such a decision.

10

Partial evaluation is a conceptual framework relevant to relate interpreters and compilers.

Can we make general statements about the performance of interpreted code vs compiled code?

Programming languages are specifications (written in some report, such as R5RS or n1570). They are not software, so it does not even make sense to speak of performance. But some programming language may have several implementations, including interpreters and compilers.

Even in traditionally compiled languages (that is, languages whose implementations are often compilers) like C, some parts are often interpreted. For example, the format control string of printf (defined in the C standard) is often "interpreted" (by the C standard library, which has a printf function using variable arguments techniques) but some compilers (including GCC) are able -in limited specific cases- to optimize it and "compile" it into lower-level calls.

And some implementations, even within "interpreters", are using JIT compilation techniques (so generate machine code at runtime). A good exemple is luajit. Other implementations (e.g. Python, Ocaml, Java, Parrot, Lua) are translating the source code into a bytecode which is then interpreted.

SBCL is a Common Lisp "compiler" which dynamically translates every REPL interaction (and calls to eval etc...) into machine code. So you feel it is an interpreter. Most JavaScript implementations in browsers (e.g. v8) use JIT compilation techniques.

In other words, the difference between interpreters and compilers is very fuzzy (actually there is a continuum between both), and practically speaking, most programming language implementations has often both an interpreter and a compiler (at least to byte code) facet.

An implementation can be fast or slow independently of using most "compiler" or "interpreter" like techniques.

Some language traits favor an interpreting approach (and can only be efficiently compiled thru whole program analysis).

For some types of problems, designing the software with some metaprogramming approaches is worthwhile and gives important speed-ups. You could imagine that given some specific input, your program dynamically generates specialized code to process it. This is even possible with C or C++ (either using some JIT library, or generating some C code, compiling it as a plugin which gets dynamically loaded).

See also this related question about Python, and that

7

For code like A = A + B, that can compile down to one or two machine instructions, each taking a certain number of cycles. No interpreter can do the same thing in that number of cycles for a simple reason.

The interpreter also executes an instruction set of its own (call them byte-codes, p-codes, intermediate language, whatever). Each time it sees a byte-code like ADD, it has to look it up somehow and branch to the code that does the addition.

The next time it sees it, it has to repeat that lookup, unless it has a way to remember the prior lookup. If it does have a way to remember the prior lookup, it is no longer what we call an "interpreter", but rather a just-in-time compiler, or JITter.

On The Other Hand...

For code like callSomeFunction( ... some args ...), how many cycles are spent between entering that code and leaving it? It all depends on what happens inside callSomeFunction. It could be a few, and it could be trillions, even if callSomeFunction is itself compiled. If it's a lot, there is no point in debating the interpretation cost of that line of code - the money is elsewhere.

Remember interpreted languages have a value of their own, such as, no need to compile them. (The "compilation" of surface syntax to byte codes takes trivial time. Take R or MATLAB, for example.)

Also, there is flexibility needed for intelligent levels of programming. In Minsky's Society of Mind, Chapter 6.4 B-Brains, there are A programs that deal with the world, and there are B programs that deal with A programs, and there can be further levels. Programs that write and manage other programs can be more easily done in interpretive systems.

In Lisp, you can write (+ A B) to add A and B, but once it is written you only have the choice to run it or not. You can also write (eval (list '+ 'A 'B)) which constructs the program and then executes it. It could construct something different.

The program's subject is another program. This is easier to write in an interpreted language (though, as Jörg points out, newer versions of Lisp, while they have eval, compile-on-the-fly, so they do not have the speed penalty of interpreting).

  • 1
    I find it interesting that you say writing meta-level programs is easier in interpreted languages, yet use Lisp as an example, where most implementations are compiled. – Jörg W Mittag Jan 10 '17 at 14:02
  • 1
    @JörgWMittag: Sure, but they all have eval and apply functions, which are interpreters. – Mike Dunlavey Jan 10 '17 at 14:04
  • 2
    I'n pretty sure that at least on SBCL, eval is not interpreted. And neither is apply. There certainly are implementations that contain interpreters, but SBCL does not. – Jörg W Mittag Jan 10 '17 at 14:09
  • 1
    Interpreter can optimize loop condition out in runtime and squash remaining iterations. This is rarely possibly in compiled programs. Specifically, Oracle's Hotspot does exactly that. – Basilevs Jan 10 '17 at 14:42
  • 2
    @JörgWMittag: Sure eval is not interpreted. It is an interpreter. – Mike Dunlavey Jan 10 '17 at 15:23
5

Kind of, sort of, it depends but as general rule the compiled - be it through JIT or statically compiled - environment will be faster for many compute-intensive tasks - assuming for simplicity same language.

Part of the reason is that interpreted languages need to have interpreter loop - loop which reads an instruction, selects appropriate action to take and executes it. In best case like interpreting Python or Java bytecode (as old JVM did) it have an overhead of few instructions and played havoc with branch predictor - without the last one you can expect huge penalties due to mispredictions. Even a very dumb JIT should speed this significantly.

That said the interpreted language may cheat. For example Matlab have optimized routines for matrix multiplication and with few changes you can get code running on GPU (disclaimer: I work for nVidia - any opinion expressed here is mine and does not represent view of my employer). That way you can write short and powerful higher level code without worrying about details - someone did take care about it and pour time and resources into optimizing it in low-level language. There is nothing inherited about it and it does not prevent, for example, Matlab to JIT the code but often there is no reason as overhead of calling high-level routine is minimal compared with time spent in low-level ones.

TL;DR - the compiled programs have huge performance benefits over the interpreted ones (for apples-to-apples comparison see PyPy Speed). However the speed of executable is only part of problem and it may not contribute much to overall speed (if time is mostly spent in libraries). Also the implementation matters.

5

Your assumption is well founded, although it is an assumption.

I am not going to go over the reasons why compiled code should be faster than interpretted code: if you know how computers work, that will be obvious. The difference can be orders of magnitude for certain types of problem. If your reviewer seriously disputes that general case, they don't know what they are talking about.

Where they may have a point though is whether the difference is significant in the type of application you are developing. If it is mostly I/O or mostly calling compiled libraries and it doesn't have a lot of computation, the overhead of the interpretation process may indeed be insignificant.

But the point of my post is this: as an I.T. expert you will be often called to make snap decisions based on a general knowledge of how things should work. Doing a specific test might get you a more accurate answer, but it will cost a lot more and it won't get you there first.

But from time-to-time you do get caught out. It's happened to me. You make a good assumption and then you find you failed to take into account the stupidity of the world.

But I can't explain as well as my favourite Dilbert cartoon of all time. Nothing shows better than this the perils of being a smart-ass.

alt text

TL;DR: you should be right, but check the real world just in case.

3

Unless you use something somewhat exotic, your problem won't be about performance about an interpreted language A and compiled language B.

Because if you/your team know A and not B and so write way better code in A than B you can end up having much better performance in A than B. If you have people experienced in one language and the language/libraries can does the job you need, stick to it.

Here's a link about regex in various languages; you'll see that regex are implemented better in some language even if compiled or not: http://benchmarksgame.alioth.debian.org/u64q/performance.php?test=regexdna

  • The problem is that the team can use both A and B, and did not indicate a preference for either when I asked them for input. – EpicSam Jan 10 '17 at 13:12
  • @EpicSam what about their past projects then ? – Walfrat Jan 10 '17 at 13:17
  • 1
    Both A and B have been used. However they want to move to using 1 technology for all projects. Both technologies's performance is good enough for current projects. However the potential higher performance of one of them should be considered when looking to potential future projects. – EpicSam Jan 10 '17 at 13:29
  • 3
    @EpicSam better look for the community around the language and the library/Framework available to help you in your projects. – Walfrat Jan 10 '17 at 13:43
  • 6
    I agree with @Walfrat . I would bet the language with "highest potential" is straight-up assembly or some raw CPU instruction set. But we aren't talking about those languages because its "obvious" that having tools like compilers, interpreters, and pre-written libraries is ultra important. I think the availability of tools / community knowledge is more important than the choice between interpreted/compiled – Ivan Jan 10 '17 at 13:57
1

I think it's just not a good idea to talk about performance of two technologies just based on the fact that one is compiled and the other one is interpreted. As stated in other answers, it may depend on the area of application (some languages may be optimised to do some operations very quickly and do other things more slowly) as well as on experience of people that are about to use that technology.

I don't think it's reasonable to expect that you will get a performance boost if you take some excellent interpreted language coders and give them some technology that they are unfamiliar with - maybe in theory the latter MAY result in better performance, but in reality, without necessary skills and experience, you won't use all the optimization opportunities.

From one of the well-known Silicon Valley company employees I've also heard that they prefer the language that is simpler to use as it is more costly and troublesome to pay some skilled developers to maintain a complicated, but highly optimized code than just to buy more rig in order to deal with the less efficient implementation, so that also should be considered while choosing the technology.

0

Once I had to make a similar sweeping statement to justify a big decision.

First, they may not want to believe a humble engineer, so I found some comparable benchmark tests and quoted them. There are a lot of them about, from people like Microsoft, or renowned universities. And they will say stuff like: Method A is between 3 and 10 times faster than method B, depending on variables X and Y.

Second, you might like to run a benchmark of your own, maybe using a representative chunk of the code in question, or something similar you already have. Run it 1000 times overnight so there really is a measurable difference.

At this point the difference (or lack of it) between A and B should be so clear that you only have to present it. So format the results clearly, with diagrams if possible, stating all assumptions and defining all data used.

  • 1
    The problem with doing my own benchmark is that I would effectively just be bench-marking 2 specific programs written in A and B, not A and B as a whole. It should be possible to create programs that solves problem X in both A and B where A is faster and then rewrite them in way B is faster and vice versa. This essentially allows you to work backwards from your conclusion, for example if you prefer A then either choose a scenario in which A is faster or simple optimise the A version till it outperforms the one in B. Therefore we could only conclude performance in a specific case not general. – EpicSam Jan 12 '17 at 15:33
  • Depending on how big and important this decision is, you could implement a representative part of the functionality using both A and B. If this is a really big decision then this is not wasted time. You would have to justify how representative your section is, and that you are not trying to bias in favour of your personal preference. – RedSonja Jan 13 '17 at 9:46
  • 1
    @EpicSam Find someone clearly favoring A and let them optimize the benchmark for A. Do the same for B. You only need to ensure that both programmers are of about the same level and spend about the same time. There's still the problem of selecting the benchmark, but let both programmers be involved in the decision; ideally, let them agree on the benchmark. Another problem is the wasted time, but this can be handled by choosing something simple and useful. – maaartinus Jan 14 '17 at 18:49
0

I would argue that any dynamic language has an advantage over statically compiled ones: "Runtime optimizations"

That is one of the reasons Java can be faster than C++

Yes, loading a dynamically typed language will always have the cost of translation and be at a disadvantage. But once it is running, the interpreter can profile and enhance frequent code paths with runtime information that static languages will never have

NOTE: Well, Java is an interpreted language, not a dynamic one. But it is a great example of what you can speed up with runtime information

-3

... I also state that since a program could be written in many ways it is still possible a program written in tech A could outperform one written in tech B.

When I submitted this report for review, the reviewer stated that I offered no clear reason why in general the overhead of the interpretation process would be large enough that we could conclude that tech B's performance would be better. ...

This would be my approach:

In general, interpreters are compiled, so every interpreted technology is nothing else than a compiled technology if looked at it on a low level. Therefore, compiled technologies just are more and with more possibilities you can never get worse if you are smart (which in general you are). It depends on how much information there is available at compile time and how much information available only at runtime and how good the compilers and interpreters are, but it should theoretically always be possible to at least equal the performance of any interpreter with a suitable compiler, just because interpreters are fabricated by compilers. That it's possible, doesn't mean it's the case for your techs A and B though.

In practice, just tell the reviewer about all the benchmarks available where compiled and interpreted systems are compared. Then ask him to suggest an interpreter that beats your optimized Assembly coded specific algorithm.

One should maybe add, that any such general statement doesn't help at all when comparing two specific techs A and B. There the choice of A and B matters much, much more, than if they are interpreted or compiled.

  • 2
    Totally wrong. You do not understand how interpreters work vs how compilers work. – rghome Jan 12 '17 at 15:59

protected by gnat Jan 11 '17 at 12:48

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.