1

Developing Big Data processing pipelines and storage, you probably come across software which is more or less a part of the Hadoop ecosystem. Be it Hadoop itself, Spark/Flink, HBase, Kafka, Accumulo, etc.

Now all of these have been very well implemented, offering fast and high-quality solutions to the developers needs. Still, especially with the Big Data usage patterns in mind, a huge amount of object allocations and deallocations happen. It is probably worthwhile to use a non-garbage collected language, like C++.

Another reason I could find for myself, why Java applications are so popular in this domain, is the distributed deployment. One key characteristic of Big Data applications is the size, they don't fit on a single machine. The JVM allows really simple deployment (just copy the bytecode around). But is this really an argument? Looking at our own cluster, the hardware is quite similar and I would assume that this holds true for most companies. So even compiled machine code should be easy to move around to all machines.

For me personally, the biggest reason would probably be DRY (don't repeat yourself). It started in Java and libraries and frameworks grew around it. They work very well and nobody is willing to invest in rewriting the whole stack in a different programming language for (if at all) marginal gain.

Maybe someone of you has a deeper insight than me?

|improve this question
  • It is perhaps worth noting that Spark at least has effectively written its own memory management layer. – Philip Kendall Jan 30 at 18:52
  • 3
    Do you have profiled your workloads and have hard evidence that object allocation and deallocation is an actual bottleneck? Which garbage collector did you use? There are dozens of them, and some of them might be better for certain use cases. Note that for most modern high-performance generational copying tracing garbage collectors, allocating temporary objects is O(1), just bumping a pointer, identical to allocating a stack value in, say, C++. And garbage collecting the young generation is O(#live objects), so deallocation of short-lived objects is 100% free. – Jörg W Mittag Jan 30 at 20:09
  • 1
    My intention for this question was much more general. Because of the specific use case for Big Data processing, two important factors come into play: performance and scalability. Looking at the software out there, almost always a JVM language was chosen and I am just curious what the deciding points were. I am explicitly not arguing that Java is slower than C++. We have a bit of profiling data for our applications and under heavy workloads we can see the garbage colloctor stalling the whole pipeline, we tested different collectors, but that would be a question for SO. – flowit Jan 30 at 20:22
  • 4
    Your assumption that garbage collection is very expensive over time may not be correct. – Thorbjørn Ravn Andersen Jan 31 at 1:19
  • 2
    Guesswork: From what I can see from the engineering POV on the topics I have been involved in, I'd say Java/C#/C++/whatever "doesn't matter" from a performance POV for Big Data, because, from what I can see, the performance characteristics are dominated by the distributed nature of the problem, and getting a handle on the data sizes involved. (i.e. network/disk systems) If the actual processing of the data on the single nodes is slightly more efficient / faster / less latency, doesn't seem to be the key point. – Martin Ba Jan 31 at 12:03
6

Hadoop was originally written in Java, because it was used to "fix" problems in Nutch, which also was written in Java. Nutch, in turn, was written in Java because it was a write once run anywhere solution.

As for whether C++ or another language would have been a better choice, that's definitely up for debate. With modern architectures, I'd trust Java or C#'s garbage collector over a random developer's judgement. For most applications, we don't need to be heavily concerned with resource usage, beyond normal best practices, unlike the early days of computing where every bit was important and needed to be managed.

However, Big Data is definitely an outlier for that approach. I still would have a developer who understood how Java's garbage collection worked code in Java than trust a developer in C++ to know how to do garbage collection well.

That said, this will almost always get into a debate about Java and C# developers being spoiled by their frameworks, and as a C# developer, I'd always rather have a library written and tested by a team of professionals (or a library written and tested and used by the masses) than try to do it myself. Instead of knowing how to manually allocate memory and manage it (which I can do in C, but haven't since school) I'd rather just understand how the C# garbage collector works.

|improve this answer
0

For me, garbage collection is a solution to DRY out memory-management code. All those malloc and free calls can be automatically handled by garbage collection fairly systematically and efficiently. Sure there are some edge cases, but unless you can profile garbage collection as the most important bottleneck in your application, I don’t see why you’d bother switching to a manual memory management language like C++. At a certain point, you have to trust that libraries/frameworks/systems do what they say they do, and appropriately/efficiently.

Garbage collection systems have been extensively developed and tested. Given my own lackluster knowledge of memory management, I think I would rather trust experts who have collaborated with researchers with mathematical proofs to certain garbage collection strategies, rather than trust a developer to say “yeah I think we should free up that memory here because we won’t need those objects later”, then inevitably cause a re-allocation of those same objects later because they forgot they had already freed them.

If you have some very niche part of the application that needs very specific and detailed memory management, for example, couldn’t you just write that as a microservice or small library in the desired language, then link it back to your primary language? I don’t know what it’s called in the desktop library world, but you can certainly call C++ compiled libraries through Java if you really needed that level of micromanagement.

But I imagine that since Hadoop hasn’t done that yet (to my knowledge), memory management actually isn’t a problem for this ecosystem.

|improve this answer
  • 1
    Modern C++ heavily encourages the usage of various kinds of smart pointers to avoid the need for manual memory management. By keeping track of who owns the memory, the language can automatically deallocate it when it goes out of scope. – 8bittree Jan 31 at 17:50
  • @8bittree Fair enough, it’s been 8 years since I programmed in C/C++, and we never were really into the “current standard” as far as university education goes, so my knowledge is at least a decade out of date! – Chris Cirefice Jan 31 at 20:05
  • Using simple malloc/free is definitely considered outdated and substandard in modern C++. Do not evaluate the language on their level of risk; it's like evaluating a modern car by looking at the model T. – Aganju Feb 14 at 15:59

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.