I've read in a lot of forums, a lot of users agree C++ is faster than Java (even if it leads for a microsecond, it is important), so why are the majority of data mining software or software tools developed in java? Shouldn't there be a greater advantage by using C++??

closed as primarily opinion-based by gnat, Doc Brown, Christophe, 8bittree, Robert Harvey Sep 14 '17 at 1:15

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 36
    Come on, this topic has been beaten to death: the answer to "why does one use X if Y is faster?" is always "because speed is not the only quality criteria for software, and often not the most important one" and "because Y is not always or generally faster than X". – Doc Brown Sep 13 '17 at 5:40
  • 2
    Maybe because it's easier to write portable Java code than portable C++ code, and maybe the standard GUI and database access libraries are missing in C++ ? – Christophe Sep 13 '17 at 6:52

Assuming for a moment that the general performance difference is true...

  • Java has more mature data manipulation libraries.
  • Java has mildly sane string manipulation.
  • Java has a wider userbase.
  • Java will generally be written correctly faster.
  • ETL processes will be disk bound, so speed of your runtime doesn't matter.

...and when your processes take a few hours, nobody really cares if your save a few minutes here or there.

  • 14
    It's about bottleneck. The program's actual execution speed is so far ahead of disk IO that it becomes irrelevant. – Nelson Sep 13 '17 at 3:52
  • 5
    There are two claims here that seems rather... specious. Specifically (1) Java Strings are a half-hearted attempt at Unicode, which is not really better than not attempt at all: for ASCII it doesn't matter, and for the rest it's likely to lull the user in a false sense of "it's working" and (2) the "correctly written faster" seems dubious, while it's less likely to have memory errors, this does not mean it'll have less functional bugs (C++ type system is further advanced). I would advise removing those two claims, they detract from the others (which seem valid to me). – Matthieu M. Sep 13 '17 at 8:25
  • I did not meet any use case where Java Unicode was causing any unexpected errors, while using UTF-8 and UTF-16 with latin - east european characters, so Java Unicode support is well sufficient in lot of cases. While point on "written correctly faster" is probably depending on framework used and also languange used, since Java is not only language usable with JVM. – Piro Sep 13 '17 at 9:55
  • 1
    @MatthieuM. - have you used C++ for string manipulation? – Telastyn Sep 13 '17 at 11:18
  • @Telastyn: Yes. Of course, I use the Boost library, which contains lots of goodies (especially Boost.Algorithms). C++ does not help with Unicode either, of course, but for ASCII string manipulations it doesn't lag behind Java. – Matthieu M. Sep 13 '17 at 11:23

I think there's more to this than simply that the workflow for ETL is typically I/O bound, not CPU bound; that's just the justification.

The real issue here is development cost.

C++ applications of comparable functionality are slower to build and harder to maintain than java ones.

There are many reasons for this; poor C++ package management, low level memory management, comparatively poor tooling support, etc.

When starting a project, the question becomes:

Is the performance benefit we get from using low level code (C, ASM, C++, etc) worth the additional cost of building it using that technology?

If the answer is no, then there is no business case to build it using them.

You can see this happening in a comparable hybrid model with machine learning and python: Should tensorflow be built entirely using python? The answer is no, it would be too slow and useless. So it's built using C++ and backed with GPU compute, with a python api for ease of use.

However, for ETL products, because it is unusual for the workflow to be CPU bound, there's no meaningful case to be made for implementing in C++; that's why you also see many python ETL frameworks as well.

  • 1
    It's worth mentioning that the development cost may well not just be higher but completely out of reach. "C++ performance" doesn't just happen by magic. A naively-written C++ program has a high chance of being outright slower than many alternatives in languages with JIT and GC goodies, and making it faster may well be an extremely nontrivial undertaking. – Leushenko Sep 13 '17 at 23:43

Note: when I refer to "Java" here, it's only because that happens to be what was mentioned in the question. Substitute almost any other tool and most of the rest of the answer remains reasonably alcohol.

There are two schools of thought about questions like this. In his comment, I think @Docbrown summarizes one school of thought quite succinctly, while losing little that's of real value:

Come on, this topic has been beaten to death: the answer to "why does one use X if Y is faster?" is always "because speed is not the only quality criteria for software, and often not the most important one" and "because Y is not always or generally faster than X".

Looking at this a little more generally, the argument is basically: programmers have chosen X over Y because careful consideration of the technical requirements for the task at hand show that X is the superior tool for the task.

In short, it supposes that the people making the decisions are unaffected by preexisting biases, are immune to confirmation bias, and in general make their decisions based only on the technical considerations that apply to the task at hand.

This is a school of thought, and schools of thought are required to have names, so I'm going to title this the "grossly unrealistic" school of thought.

The second school of thought doesn't exactly ignore technical factors, but also tries to consider human factors. It assumes that the people involved in making such decisions are actual...people. They have preexisting biases. They are affected by confirmation bias.

This doesn't mean that they necessarily make bad decisions, or that their reasons for making those decisions are any less valid--but with any attempt at being realistic, we quickly realize that technical factors are rarely the sole criteria used for making such decisions--in fact, more often than not, technical factors are likely to play a fairly minor role in such decisions.

Under network effects, we consider that if (for example) a company uses a database from a company (e.g., IBM or Oracle) that favors Java, they're pretty likely to find lots of advice pushing them toward using Java. If, on the other hand, they used (for example) Microsoft SQL server, there's a lot better chance that they might use something based on .NET for their TL tasks. On the third hand (so to speak) if you're looking at a small startup with little or no commitment to existing software stacks, chances are pretty decent that they're going to use things like Python and Pandas (and chances are they don't call it something "dignified" like "ETL"--they more likely just call it a "data scraper" or something on that order).

Looking at existing personnel, we get two entirely different considerations, though they tend to work in the same direction. One is a matter of existing biases and confirmation bias. If you have an IT department full of people who write Java (as many large enterprises do) chances are that at least some of them actually prefer to use Java. Unless it's immediately obvious that Java is much worse than the alternative, they're likely to consider it the first choice. Confirmation bias then plays its part: for example, if they Google for information about their best choice, from their viewpoint nearly every article they find will be seen as evidence that Java is really the best choice1.

The second consideration is much easier to defend: if we have an IT department full of people who know Java, it's a lot better to add a few more people who know Java to handle this new task, even if (under other circumstances) some other tool would really be a better fit for the job. Even if some other tool would be superior for this particular task, the savings for a particular task often won't justify the overhead of introducing new tools, hiring people the current personnel can't really evaluate well, following new procedures, etc.

There's yet another point that's also worth considering. In a lot of cases, your view of what's popular is shaped heavily by your background. Many of the same factors as above (and more besides) can come into play in reinforcing this view. One that's not mentioned above, but seems clearly relevant to me is simply terminology. If you Google for something like "database ETL", your results will be dominated by results related to Oracle and Java (with, probably, a few from other "Enterprise" oriented vendors like SAS). If, instead, you search for something like "database scraper", you see a view of the world in which Oracle and Java basically don't exist.

1. Yes, even if the article actually very clearly says something like "we found Java clearly inferior for this task", confirmation bias will allow many people to (apparently honestly) find something like one outlier where Java performed well, and immediately be convinced that it's the think that really matters for their task.

  • While I like this answer in principle, it doesn't have much to do with the question that was asked. Tomes like this properly belong on a blog site, or perhaps meta. – Robert Harvey Sep 14 '17 at 1:17

As an aside many of the data mining applications are written in Python, sometimes with Java used in data display.

However the following points to consider apply to both python and java:

  1. Both Java & Python can be written once and run, unchanged, on a wide variety of machines so you can develop and test on a Windows machine and then use on a super computer if necessary.
  2. Both have a great many mature libraries for things like data display, data gathering (e.g. web scraping) and database interaction.
  3. In most cases these libraries are available freely with little or no cost and with the source code available.
  4. The libraries are also available on most/all platforms.
  5. In both cases the entire tool chain is available freely.
  6. Both scale well regarding data size.

Contrast this with C++ where the tool chain costs, sometimes a considerable amount of money. Then if you need to talk to a database you may well need to buy or write a connector library and that library may well only work on a specific platform and may be limited to a single installation.

Not the answer you're looking for? Browse other questions tagged or ask your own question.