I've read in a lot of forums, a lot of users agree C++ is faster than Java (even if it leads for a microsecond, it is important), so why are the majority of data mining software or software tools developed in java? Shouldn't there be a greater advantage by using C++??
closed as primarily opinion-based by gnat, Doc Brown, Christophe, 8bittree, Robert Harvey♦ Sep 14 '17 at 1:15
Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.
Assuming for a moment that the general performance difference is true...
- Java has more mature data manipulation libraries.
- Java has mildly sane string manipulation.
- Java has a wider userbase.
- Java will generally be written correctly faster.
- ETL processes will be disk bound, so speed of your runtime doesn't matter.
...and when your processes take a few hours, nobody really cares if your save a few minutes here or there.
I think there's more to this than simply that the workflow for ETL is typically I/O bound, not CPU bound; that's just the justification.
The real issue here is development cost.
C++ applications of comparable functionality are slower to build and harder to maintain than java ones.
There are many reasons for this; poor C++ package management, low level memory management, comparatively poor tooling support, etc.
When starting a project, the question becomes:
Is the performance benefit we get from using low level code (C, ASM, C++, etc) worth the additional cost of building it using that technology?
If the answer is no, then there is no business case to build it using them.
You can see this happening in a comparable hybrid model with machine learning and python: Should tensorflow be built entirely using python? The answer is no, it would be too slow and useless. So it's built using C++ and backed with GPU compute, with a python api for ease of use.
However, for ETL products, because it is unusual for the workflow to be CPU bound, there's no meaningful case to be made for implementing in C++; that's why you also see many python ETL frameworks as well.
Note: when I refer to "Java" here, it's only because that happens to be what was mentioned in the question. Substitute almost any other tool and most of the rest of the answer remains reasonably alcohol.
There are two schools of thought about questions like this. In his comment, I think @Docbrown summarizes one school of thought quite succinctly, while losing little that's of real value:
Come on, this topic has been beaten to death: the answer to "why does one use X if Y is faster?" is always "because speed is not the only quality criteria for software, and often not the most important one" and "because Y is not always or generally faster than X".
Looking at this a little more generally, the argument is basically: programmers have chosen X over Y because careful consideration of the technical requirements for the task at hand show that X is the superior tool for the task.
In short, it supposes that the people making the decisions are unaffected by preexisting biases, are immune to confirmation bias, and in general make their decisions based only on the technical considerations that apply to the task at hand.
This is a school of thought, and schools of thought are required to have names, so I'm going to title this the "grossly unrealistic" school of thought.
The second school of thought doesn't exactly ignore technical factors, but also tries to consider human factors. It assumes that the people involved in making such decisions are actual...people. They have preexisting biases. They are affected by confirmation bias.
This doesn't mean that they necessarily make bad decisions, or that their reasons for making those decisions are any less valid--but with any attempt at being realistic, we quickly realize that technical factors are rarely the sole criteria used for making such decisions--in fact, more often than not, technical factors are likely to play a fairly minor role in such decisions.
Under network effects, we consider that if (for example) a company uses a database from a company (e.g., IBM or Oracle) that favors Java, they're pretty likely to find lots of advice pushing them toward using Java. If, on the other hand, they used (for example) Microsoft SQL server, there's a lot better chance that they might use something based on .NET for their TL tasks. On the third hand (so to speak) if you're looking at a small startup with little or no commitment to existing software stacks, chances are pretty decent that they're going to use things like Python and Pandas (and chances are they don't call it something "dignified" like "ETL"--they more likely just call it a "data scraper" or something on that order).
Looking at existing personnel, we get two entirely different considerations, though they tend to work in the same direction. One is a matter of existing biases and confirmation bias. If you have an IT department full of people who write Java (as many large enterprises do) chances are that at least some of them actually prefer to use Java. Unless it's immediately obvious that Java is much worse than the alternative, they're likely to consider it the first choice. Confirmation bias then plays its part: for example, if they Google for information about their best choice, from their viewpoint nearly every article they find will be seen as evidence that Java is really the best choice1.
The second consideration is much easier to defend: if we have an IT department full of people who know Java, it's a lot better to add a few more people who know Java to handle this new task, even if (under other circumstances) some other tool would really be a better fit for the job. Even if some other tool would be superior for this particular task, the savings for a particular task often won't justify the overhead of introducing new tools, hiring people the current personnel can't really evaluate well, following new procedures, etc.
There's yet another point that's also worth considering. In a lot of cases, your view of what's popular is shaped heavily by your background. Many of the same factors as above (and more besides) can come into play in reinforcing this view. One that's not mentioned above, but seems clearly relevant to me is simply terminology. If you Google for something like "database ETL", your results will be dominated by results related to Oracle and Java (with, probably, a few from other "Enterprise" oriented vendors like SAS). If, instead, you search for something like "database scraper", you see a view of the world in which Oracle and Java basically don't exist.
1. Yes, even if the article actually very clearly says something like "we found Java clearly inferior for this task", confirmation bias will allow many people to (apparently honestly) find something like one outlier where Java performed well, and immediately be convinced that it's the think that really matters for their task.
As an aside many of the data mining applications are written in Python, sometimes with Java used in data display.
However the following points to consider apply to both python and java:
- Both Java & Python can be written once and run, unchanged, on a wide variety of machines so you can develop and test on a Windows machine and then use on a super computer if necessary.
- Both have a great many mature libraries for things like data display, data gathering (e.g. web scraping) and database interaction.
- In most cases these libraries are available freely with little or no cost and with the source code available.
- The libraries are also available on most/all platforms.
- In both cases the entire tool chain is available freely.
- Both scale well regarding data size.
Contrast this with C++ where the tool chain costs, sometimes a considerable amount of money. Then if you need to talk to a database you may well need to buy or write a connector library and that library may well only work on a specific platform and may be limited to a single installation.