Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
Sometimes Java outperforms C++ in benchmarks. Of course, sometimes C++ outperforms.
See the following links:
But how is this even possible? It boggles my mind that interpreted bytecode could ever be faster than a compiled language.
Can someone please explain? Thanks!
First, most JVMs include a compiler, so "interpreted bytecode" is actually pretty rare (at least in benchmark code -- it's not quite as rare in real life, where your code is usually more than a few trivial loops that get repeated extremely often).
Second, a fair number of the benchmarks involved appear to be quite biased (whether by intent or incompetence, I can't really say). Just for example, years ago I looked at some of the source code linked from one of the links you posted. It had code like this:
init0 = (int*)calloc(max_x,sizeof(int));
init1 = (int*)calloc(max_x,sizeof(int));
init2 = (int*)calloc(max_x,sizeof(int));
for (x=0; x<max_x; x++) {
init2[x] = 0;
init1[x] = 0;
init0[x] = 0;
}
Since calloc provides memory that's already zeroed, using the for loop to zero it again is obviously useless. This was followed (if memory serves) by filling the memory with other data anyway (and no dependence on it being zeroed), so all the zeroing was completely unnecessary anyway. Replacing the code above with a simple malloc (like any sane person would have used to start with) improved the speed of the C++ version enough to beat the Java version (by a fairly wide margin, if memory serves).
Consider (for another example) the methcall benchmark used in the blog entry in your last link. Despite the name (and how things might even look), the C++ version of this is not really measuring much about method call overhead at all. The part of the code that turns out to be critical is in the Toggle class:
class Toggle {
public:
Toggle(bool start_state) : state(start_state) { }
virtual ~Toggle() { }
bool value() {
return(state);
}
virtual Toggle& activate() {
state = !state;
return(*this);
}
bool state;
};
The critical part turns out to be the state = !state;. Consider what happens when we change the code to encode the state as an int instead of a bool:
class Toggle {
enum names{ bfalse = -1, btrue = 1};
const static names values[2];
int state;
public:
Toggle(bool start_state) : state(values[start_state])
{ }
virtual ~Toggle() { }
bool value() { return state==btrue; }
virtual Toggle& activate() {
state = -state;
return(*this);
}
};
This minor change improves the overall speed by about a 5:1 margin. Even though the benchmark was intended to measure method call time, in reality most of what it was measuring was the time to convert between int and bool. I'd certainly agree that the inefficiency shown by the original is unfortunate -- but given how rarely it seems to arise in real code, and the ease with which it can be fixed when/if it does arise, I have a difficult time thinking of it as meaning much.
In case anybody decides to re-run the benchmarks involved, I should also add that there's an almost equally trivial modification to the Java version that produces (or at least at one time produced -- I haven't re-run the tests with a recent JVM to confirm they still do) a fairly substantial improvement in the Java version as well. The Java version has an NthToggle::activate() that looks like this:
public Toggle activate() {
this.counter += 1;
if (this.counter >= this.count_max) {
this.state = !this.state;
this.counter = 0;
}
return(this);
}
Changing this to call the base function instead of manipulating this.state directly gives quite a substantial speed improvement (though not enough to keep up with the modified C++ version).
So, what we end up with is a false assumption about interpreted byte codes vs. some of the worst benchmarks (I've) ever seen. Neither is giving a meaningful result.
My own experience is that with equally experienced programmers paying equal attention to optimizing, C++ will beat Java more often than not -- but (at least between these two), the language will rarely make as much difference as the programmers and design. The benchmarks being cited tell us more about the (in)competence/(dis)honesty of their authors than they do about the languages they purport to benchmark.
[Edit: As implied in one place above but never stated as directly as I probably should have, the results I'm quoting are those I got when I tested this ~5 years ago, using C++ and Java implementations that were current at that time. I haven't rerun the tests with current implementations. A glance, however, indicates that the code hasn't been fixed, so all that would have changed would be the compiler's ability to cover up the problems in the code.]
If we ignore the Java examples, however, it is actually possible for interpreted code to run faster than compiled code (though difficult and somewhat unusual).
The usual way this happens is that the code being interpreted is much more compact than the machine code, or it's running on a CPU that has a larger data cache than code cache.
In such a case, a small interpreter (e.g., the inner interpreter of a Forth implementation) may be able to fit entirely in the code cache, and the program it's interpreting fits entirely in the data cache. The cache is typically faster than main memory by a factor of at least 10, and often much more (a factor of 100 isn't particularly rare any more).
So, if the cache is faster than main memory by a factor of N, and it takes fewer than N machine code instructions to implement each byte code, the byte code should win (I'm simplifying, but I think the general idea should still be apparent).
Hand rolled C/C++ done by an expert with unlimited time is going to be at least as fast or faster than Java. Ultimately, Java itself is written in C/C++ so you can of course do everything Java does if you are willing to put in enough engineering effort.
In practice however, Java often executes very fast for the following reasons:
At the same time C/C++ also have some advantages:
Overall:
All things being equal, you could say: no, Java should never be faster. You could always implement Java in C++ from scratch and thereby get at least as good performance. In practice, however:
As an aside, this reminds me of the debate regarding C in the 80s / 90s. Everyone was wondering "can C ever be as fast as assembly?". Basically, the answer was: no on paper, but in reality the C compiler created more efficient code than 90% of the assembly programmers (well, once it matured a bit).
The Java runtime isnt interpreting bytecode. Rather, it uses whats called Just In Time Compilation. Basically, as the program is run, it takes bytecode and converts it into native code optimized for the particular CPU.
But allocation is only half of memory management -- deallocation is the other half. It turns out that for most objects, the direct garbage collection cost is -- zero. This is because a copying collector does not need to visit or copy dead objects, only live ones. So objects that become garbage shortly after allocation contribute no workload to the collection cycle.
...
JVMs are surprisingly good at figuring out things that we used to assume only the developer could know. By letting the JVM choose between stack allocation and heap allocation on a case-by-case basis, we can get the performance benefits of stack allocation without making the programmer agonize over whether to allocate on the stack or on the heap.
http://www.ibm.com/developerworks/java/library/j-jtp09275/index.html
Posted by Tim Holloway on JavaRanch:
Here's a primitive example: Back when machines operated in mathematically-determined cycles, a branch instruction typically had 2 different timings. One for when the branch was taken, one for when the branch wasn't taken. Usually, the no-branch case was faster. Obviously, this meant that you could optimize logic based on the knowledge of which case was more common (subject to the constraint that what we "know" isn't always what's actually the case).
JIT recompilation takes this one step further. It monitors the actual real-time usage, and flips the logic based on what actually is the most common case. And flip it back again if the workload shifts. Statically-compiled code can't do this. That's how Java can sometimes out-perform hand-tuned assembly/C/C++ code.
Source: http://www.coderanch.com/t/547458/Performance/java/Ahead-Time-vs-Just-time
While a completely optimized Java program will seldom beat a completely optimized C++ program, differences in things like memory management can make a lot of algorithms idiomatically implemented in Java faster than the same algorithms idiomatically implemented in C++.
As @Jerry Coffin pointed out, there are a lot of cases where simple changes can make the code much faster -- but often it can take too much unclean tweaking in one language or the other for the performance improvement to be worthwhile. That's probably what you'd see in a good benchmark that shows Java doing better than C++.
Also, though usually not all that significant, there are some performance optimization that a JIT language like Java can do that C++ can't. The Java runtime can include improvements after the code has been compiled, which means that the JIT can potentially produce optimized code to take advantage of new (or at least different) CPU features. For this reason, a 10 year old Java binary might potentially outperform a 10 year old C++ binary.
Lastly, complete type safety in the bigger picture can, in very rare cases, offer extreme performance improvements. Singularity, an experimental OS written almost entirely in a C#-based language, has much faster interprocess communication and multitasking due to the fact that there's no need for hardware process boundaries or expensive context switches.
That is because the final step generating machine code happens transparently inside the JVM when running your Java program, instead of explicit when building your C++ proram.
You should consider the fact that modern JVM's spend quite a lot of time compiling the byte code on the fly to native machine code to make it as fast as possible. This allow the JVM to do all kinds of compiler tricks that can be even better by knowing the profiling data of the program being run.
Just such a thing as automatically inlining a getter, so that a JUMP-RETURN is not needed to just get a value, speeds up things.
However, the thing that really has allowed fast programs is better cleaning up afterwards. The garbage collection mechanism in Java is faster than the manual malloc-free in C. Many modern malloc-free implementations use a garbage collector underneath.
Short answer - it is not. Forget it, the topic is as old as fire or wheel. Java or .NET is not and will not be faster than C/C++. It's fast enough for most tasks where you don't need to think about optimization at all. Like forms and SQL processing, but that's where it ends.
For benchmarks, or small apps written by incompetent developers yes, the end result will be that Java/.NET is probably going to be close and maybe even faster.
In reality, simple things like allocating memory on stack, or simply using memzones will simply kill the Java/.NET on the spot.
Garbage collected world is using sort of memzone with all the accounting. Add memzone to C and C will be faster right there on the spot. Especially for those Java vs. C "high-performance code" benchmarks, that go like this:
for(...)
{
alloc_memory//Allocating heap in a loop is verrry good, in't it?
zero_memory//Extra zeroing, we really need it in our performance code
do_stuff//something like memory[i]++
realloc//This is lovely speedup
strlen//loop through all memory, because storing string length is soo getting old
free//Java will do that outside out timing loop, but oh well, we're comparing apples to oranges here
}//loop 100000 times
Try to use stack based variables in C/C++ (or placement new), they translate into sub esp, 0xff, it's a single x86 instruction, beat that with Java - you can't...
Most of the time I see those benches where Java against C++ are compared it causes me to go like, wth? Wrong memory allocation strategies, self-growing containers without reserves, multiple new's. This is not even close to performance oriented C/C++ code.
Also a good read: https://days2011.scala-lang.org/sites/days2011/files/ws3-1-Hundt.pdf
The reality is they are both just high level assemblers that do exactly what the programmer tells them to, exaclty how the programmer tells them to in the exact order the programmer tells them. The performance differences are so small as to be inconsequential to all practical purposes.
The language is not "slow", the programmer wrote a slow program. Very rarely will a program written the best way in one language outperfrom (to any practical purpose) a program doing the same thing using the best way of the alternate language, unless the author of the study is out to grind his particular axe.
Obviously if you are going to a rare edge case like hard realtime embedded systems, the language choice may make a difference, but how often is this the case? and of those cases, how often is the correct choice not blindly obvious.
See the following links ... But how is this even possible? It boggles my mind that interpreted bytecode could ever be faster than a compiled language.
Keith Lea tells you there are "obvious flaws" but does nothing about those "obvious flaws". Back in 2005 those old tasks were discarded and replaced by the tasks now shown in the benchmarks game.
Keith Lea tells you he "took the benchmark code for C++ and Java from the now outdated Great Computer Language Shootout and ran the tests" but actually he only shows measurements for 14 out of 25 of those outdated tests.
Keith Lea now tells you he wasn't trying to prove anything with the blog post seven years before, but back then he said "I was sick of hearing people say Java was slow, when I know it's pretty fast..." which suggests back then there was something he was trying to prove.
Christian Felde tells you "I didn’t create the code, just re-ran the tests." as-if that absolved him from any responsibility for his decision to publicise measurements of the tasks and programs Keith Lea selected.
Do measurements of even 25 tiny tiny programs provide definitive evidence?
Those measurements are for programs run as "mixed mode" Java not interpreted Java - "Remember how HotSpot works." You can easily find out how well Java runs "interpreted bytecode", because you can force Java to only interpret bytecode - simply time some Java programs run with and without the -Xint option.
Maybe research we recently did, could also give some ideas on how Java could be faster than C++ (in some cases). The article can be found here: https://www.researchgate.net/publication/352705416_Improving_productivity_in_large_scale_testing_at_the_compiler_level_by_changing_the_intermediate_language_from_C_to_Java
In this research project, we extended a compiler to use Java as an additional intermediate language, besides C/C++. We kept as much of the architecture as Java allowed. For the exact same input (source code) the build produced executables which read the exact same configuration files, produced the exact same functional behavior, and generated the exact same logs.
In short: we have observed, that the C/C++ implementations started up faster. But with time, the Java side implementation became gradually faster, to the point where, in long-running scenarios, it could end up finishing earlier.
It would be reasonable to believe, that the Just In Time (JIT) compiler of the JVM was able to notice some optimization opportunities during execution, that were either missed by the C/C++ compiler used or only present themselves during execution. Please note, that JIT compilation itself, looking for optimization opportunities, might also contribute some runtime performance penalty to the program, as it takes away some resources from the actual work. In the situation, we observed this work seemed to bring more benefits in the long run. We hope in a future research project we can get to study this effect deeper.
Actually, JIT compilation has one major advantage compared to ahead-of-time (AOT) compilation. That is that the JIT compiler is able to optimize the machine code based on what the code is actually doing, not just what it could do.
A good example is a virtual function call (method call in Java). When you write (C++) pBase->SomeVirtualFunction(); it may call either Derived1::SomeVirtualFunction or Derived2::SomeVirtualFunction. In C++, the compiler will emit instructions to check the object type and call the appropriate function. (This will take the form of an indirect branch through a vtable)
In Java, the JIT compiler is able to notice that you're always calling Derived1::SomeVirtualFunction - and just call it directly. This is called inline caching (for Java see here). The benefit is that the CPU already knows which function is being called and doesn't have to wait for a memory access or indirect branch to resolve. The compiler can even inline the function or specialize it.
Of course, it also has to insert code to make sure this was the correct function to call - if not, it has to undo whatever the code did up to that point, then go back and recompile the code with a proper virtual call and resume it. The check can be allowed to run in parallel with the actual code - think of it as speculative execution on a software level.
However, this is unlikely to affect benchmarks, as benchmarks are written for optimal performance in each language. If virtual calls are slow in C++, the benchmark will be written to avoid virtual calls.
