I am not a professional coder, but rather an engineer/mathematician that uses computer to solve numerical problems. So far most of my problems are math-related, such as solving large scale linear systems, perform linear algebra operations, fft, mathematical optimization etc.. At smaller scale, these are best handled by Matlab (and similar high-level math packages). At large scale, however, I have to resort to old-fashioned (and VERY TEDIOUS) lower-level languages such as Fortran or C/C++. That is not all, when the problem become excessively large, parallelization becomes necessary, and MPI in that regard is a nightmare to wrestle through.
I have been hearing about hadoop and "big data" recently on a regular basis. But my impression was hadoop is NOT a general-purpose parallel computing paradigm, in the sense that it handles data that need minimal mutual communications (correct me if I am wrong). For example, in a general LA operation on a large data set distributed among many nodes, rather than processing one's own piece of data independently, each node must communicate with EVERY other node to send/get info before it's own data can be correctly processed, and the whole data set is updated only after ALL communications have been done. I hope I made my point clear: in numerical applications, data-processing is naturally GLOBAL, and the "graph" for communication often is a FULL GRAPH (though there ARE exceptions).
My question is, is hadoop suitable for such purpose, or is it only designed for "simple" data processing jobs, where communications between the distributed nodes are sparse?