Why is the whole Hadoop ecosystem written in Java?

Question

Developing Big Data processing pipelines and storage, you probably come across software which is more or less a part of the Hadoop ecosystem. Be it Hadoop itself, Spark/Flink, HBase, Kafka, Accumulo, etc.

Now all of these have been very well implemented, offering fast and high-quality solutions to the developers needs. Still, especially with the Big Data usage patterns in mind, a huge amount of object allocations and deallocations happen. It is probably worthwhile to use a non-garbage collected language, like C++.

Another reason I could find for myself, why Java applications are so popular in this domain, is the distributed deployment. One key characteristic of Big Data applications is the size, they don't fit on a single machine. The JVM allows really simple deployment (just copy the bytecode around). But is this really an argument? Looking at our own cluster, the hardware is quite similar and I would assume that this holds true for most companies. So even compiled machine code should be easy to move around to all machines.

For me personally, the biggest reason would probably be DRY (don't repeat yourself). It started in Java and libraries and frameworks grew around it. They work very well and nobody is willing to invest in rewriting the whole stack in a different programming language for (if at all) marginal gain.

Maybe someone of you has a deeper insight than me?

Answer 1

Hadoop was originally written in Java, because it was used to "fix" problems in Nutch, which also was written in Java. Nutch, in turn, was written in Java because it was a write once run anywhere solution.

As for whether C++ or another language would have been a better choice, that's definitely up for debate. With modern architectures, I'd trust Java or C#'s garbage collector over a random developer's judgement. For most applications, we don't need to be heavily concerned with resource usage, beyond normal best practices, unlike the early days of computing where every bit was important and needed to be managed.

However, Big Data is definitely an outlier for that approach. I still would have a developer who understood how Java's garbage collection worked code in Java than trust a developer in C++ to know how to do garbage collection well.

That said, this will almost always get into a debate about Java and C# developers being spoiled by their frameworks, and as a C# developer, I'd always rather have a library written and tested by a team of professionals (or a library written and tested and used by the masses) than try to do it myself. Instead of knowing how to manually allocate memory and manage it (which I can do in C, but haven't since school) I'd rather just understand how the C# garbage collector works.

Answer 2

The comment that "performance" is an important factor is odd, because "performance" is not a single monolithic thing.

The two most obvious different types of "performance" constraint are:

1. Throughput (long-running servers, jobs that take a long time to complete, etc.)

   GC is typically good for this, since any overhead is amortized.

2. Latency (programs that need to start up quickly, short-running event handlers that should complete quickly, etc.)

   GC can be problematic for this, since it may take a while to reach a steady state, or we may otherwise be interested in time periods over which collection overhead cannot be amortized.

When I say collection overhead, I don't really mean "overhead compared to manual memory management", since that would imply GC is always slower. I mean just the cost of running a garbage collector at all, which can be unevenly distributed in time (compared to manual memory management which one might expect to be interleaved fairly uniformly - or at least consistently - with the program logic).

Anyway, if you think latency is more important for "Big Data processing pipelines", you should absolutely consider re-writing Hadoop. Personally, I suspect throughput is more important, and that GC is a perfectly rational choice for this.

Answer 3

For me, garbage collection is a solution to DRY out memory-management code. All those malloc and free calls can be automatically handled by garbage collection fairly systematically and efficiently. Sure there are some edge cases, but unless you can profile garbage collection as the most important bottleneck in your application, I don’t see why you’d bother switching to a manual memory management language like C++. At a certain point, you have to trust that libraries/frameworks/systems do what they say they do, and appropriately/efficiently.

Garbage collection systems have been extensively developed and tested. Given my own lackluster knowledge of memory management, I think I would rather trust experts who have collaborated with researchers with mathematical proofs to certain garbage collection strategies, rather than trust a developer to say “yeah I think we should free up that memory here because we won’t need those objects later”, then inevitably cause a re-allocation of those same objects later because they forgot they had already freed them.

If you have some very niche part of the application that needs very specific and detailed memory management, for example, couldn’t you just write that as a microservice or small library in the desired language, then link it back to your primary language? I don’t know what it’s called in the desktop library world, but you can certainly call C++ compiled libraries through Java if you really needed that level of micromanagement.

But I imagine that since Hadoop hasn’t done that yet (to my knowledge), memory management actually isn’t a problem for this ecosystem.

Answer 4

tl;dr: People are already rewriting entire swaths of the big data ecosystem in C++. If you aren't thinking of rewriting your code in C++ or Rust, you should.

You are already seeing applications completely re-written from the ground up, moving from Java to C++. A sample listing:

• Scylla, a rewrite of Apache Cassandra in C++
• Redpanda, a rewrite of Apache Kafka in C++
• Microsoft Delta Engine, a rewrite of Apache Spark in C++

But this revolution won't be limited to C++. You will also begin to see people porting code from Java to Rust.

The whole purpose for which Java was developed — to write-once, run-anyway — makes little sense when the world has moved from a heterogenous Unix/WindowsNT/Linux/MacOS universe of the 1990s into which it was born, to a modern "Linux everywhere" server market of the 2020s.

I'd argue that C++ right now is the solid bet. Rust for those who like to place bets on the dark horse. But Java? Even with the new modern JVMs and garbage collectors, it's an artifact of a paradigm that just doesn't exist in on-prem datacenters or in the cloud.

Java may still make some sense for client applications which need to run on Windows, Mac OS X and Linux, but for the servers? Even Microsoft Azure is majority Linux now.

Historically, even by the early 2010s there was little reason to put all our big data eggs in a Java basket. But software programming is a behavior. It takes years or even decades to shift the prejudices and biases of minds and the motor memory of typing fingers.

Java was used in the 2010s because it's what people learned over the preceding few decades. Yet it was already an anachronism by then. Now, with servers sporting 50-100+ CPUs, it just a really, really obvious anachronism.

Let's look at those core counts, especially. People need to do more than just simple "ports" of applications. They need to reconsider re-writing from the ground up for a world of multi-core, multi-CPU NUMA architecture servers.

They need to adopt highly asynchronous programming models. Design for more autonomy between processes than typical multi-threading paradigms. Otherwise cross-CPU traffic is going to lead to noisy contention, locks and stalls.

Beyond that, you can break away from the "virtual machine" sandbox thinking to take full advantage of every underlying hook your operating system can deliver. If you can squeeze 2% or 5% more out of your box here-and-there, you will. Because big data systems are, or should be, designed to be utterly "greedy."

Moving from Java to C++ means you can look at native OS capabilities in your programming, like io_uring and eBPF in Linux. Most developers have yet to embrace and integrate such capabilities into their application designs and their day-to-day thinking.

C++ (or Rust) also means you will need a deeper level of application developer and system architect than a typical Java body shop. You may need some hired guns that really grok the modern OS and hardware engines they are working on. Will this be beyond many people's comfort zones? Hell yes. And it's why some companies are hesitant to even go there. They know the technical depth of their bench just isn't there.

For those who are used to writing lowest-common denominator highest portability code, this is going to be a challenge. Like learning how to put the pedal to the metal on the unbounded Autobahn when they've lived their whole lives in a gated community with a 25 kph speed limit. You need to have a gut check: do you have a thrill for that kind of speed? Some people will simply retreat back to the gated community. It's safe. It's understood.

As another example, thinking ahead now a few years: we are on the verge of a revolution of byte-addressable solid state storage when persistent memory will be broadly available in public clouds. Rather than reading multi-kilobyte "disk pages," which are an artifact of the rotating media past. When that happens, people will wake up and realize: "Why are you still fetching 4k blocks? That makes no sense!" Once we get to byte-addressable persistent storage, you will need to totally rethink read and write paths, data caching, disk optimization and so on.

Which do you think is going to give you access to that underlying efficiency in current and emergent OS and hardware capability sooner? C++, Rust or Java?

(Full disclosure: I work at ScyllaDB.)