Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
I believe a lot of Fortran is used in HPC, but not sure if that's only for legacy reasons.
Features of modern programming languages like garbage collection or run-time polymorphism are not suitable for HPC since speed matters so not sure where C# or Java or C++ come in.
Any thoughts?
I have seen a lot of Java used for HPC in areas where (1) there is little legacy code, and (2) development time and code quality matter. Typical application domains are finance, data mining or bio-informatics.
It really depends on the application (there is life outside linear algebra), but the performance of recent JVMs is often on par with C code. Sometimes faster when the JVM is able to perform at runtime clever optimisations that static compilers (C, Fortran) cannot do. And definitely faster when there's a lot of symbolic computation.
Given a fixed amount of time for program development, the resulting Java code is consistently faster than C code. HPC in Java definitely makes sense when code is developed or modified frequently. Another important feature is code mobility over different hardware.
You'll find references in http://ateji.blogspot.com/2010/09/java-for-high-performance-computing.html
Regarding the Fortran assumption that two address are unique, we're working on a static analysis tool that will enable similar optimizations for code in high-level languages, but without the "Bad Things May Happen" bit. Contact me if interested.
In my years of experience, up to 5 years ago, it has always been Fortran and C. Which one depended mostly on whether the people came more from engineering or more from CS school of thought (I don't know how to put this better, okey? :-)
In what we were doing Fortran was almost exclusively used.
From what I read around nowadays, with the new updates to the Standard F2003/08 and with the introduction of Co-Arrays, it seems to be gaining momentum again.
Also, one, if not somewhat biased article - The Ideal HPC Programming Language
I think for real pedal to the metal, the only real choice is Fortran. The reasoning is that the most important thing for the exploitation of low level ILP (Instruction Level Parallism) is memory address disambiguation. The defacto rules in Fortran allow the compiler to determine that two address are unique (and hence the order of loads and store, or even stores and stores can be interchanged without risk of generating incorrect code). C leaves too much scope for overlapping pointers for the compiler to extract as much low level parallelism from the code.
Also, array alignment, w.r.t cache lines, and SSE/AVX boundaries is important to the generation and execution of efficient loops. If arrays are passed via common blocks, the compiler/loader can assure that all arrays start on the same address alignment boundaries, and more efficient SSE/AVX loads and stores can be utilized. The newer hardware can handle unaligned memory accesses, but because the memory access isn't aligned properly partial use of cache lines results in lower performance. Even if a C programmer properly aligns all his arrays, is there a mechanism to communicate this to the compiler?
To summarize, the two most important issues, are the independence of memory addresses, and the recognition by the compiler that the accessed data structures have the same "natural" alignment that the hardware wants. So far Fortran does the best job on those two tasks.
Just some anecdotal note. I haven't done any high performance computing myself.
For calculations (number crunching), Fortran and C. Yes it is for legacy reasons:
The current trend for number crunching is to write program generators that automate the source code tweaking in order to optimize performance given the cluster characteristics. These generators often output in C.
A second trend is to write in some specialized dialect of C for specific GPUs or Cell BE.
For non-numerical work, such as programs that process data from a database (but not the database itself), it is much cheaper to run on clusters of "commodity" machines without the expensive customized networking equipments. This is usually called "High Throughput Computing". And Python is the #1 language here (using the famous Map Reduce). Prior to Python, batch processing projects can be written in any language, and are usually dispatched by Condor.
Fortran, for some good and some not-so-good reasons. For heavy math crunching, a good reason is there are extensive libraries (BLAS, LAPACK) of tried-and-true subroutines, all written in Fortran (though those can be called from C and C++).
A not-so-good reason is the supposed performance advantage of Fortran over C/C++. Optimizers are pretty good, and few people understand that the benefit of optimizing a piece of code is proportional to the percent of time it is busy, which in almost all code is almost zero.
Another not-so-good reason is a culture gap between CS and non-CS programmers. Scientific programmers tend to be taught bad habits in Fortran, and to look down on the CS programmers and the bad habits they have been taught, and who look down on the former.
I've been working on some VERY calculation-intensive code in (gasp!) C#.
I'm building a GPGPU implementation of FDTD for optical modeling. On a small (128 processor) cluster, many of our simulations take weeks to run. The GPU implementations, however, tend to run about 50x faster - and that's on a consumer-grade NVidia card. We now have a server with two GTX295 dual-processor cards (several hundred cores), and are getting some Teslas soon.
How does this pertain to your language? In the same way that the C++ FDTD code we were using before was CPU-bound, these are GPU-bound, so the (very small) horsepower difference of managed vs native code doesn't ever come into play. The C# app acts as a conductor - loading OpenCL kernels, passing data to and from the GPUs, providing the user interface, reporting, etc. - all tasks that are a pain in the ass in C++.
In years past, the performance difference between managed and unmanaged code was significant enough that it was sometimes worth putting up with C++'s terrible object model to get the extra few percent of speed. These days, the development cost of C++ vs C# far outweighs the benefits for most applications.
Also, most of your performance difference isn't going to come from your choice of language, but from the skill of your developer. A few weeks ago, I moved a single division operation from the inside of a triple-nested (3D array traversal) loop, which reduced execution time for a given computational domain by 15%. That's a result of processor architecture: division is slow, which is one of those faces that you just need have picked up somewhere.
Fortran is most common, primarily due to legacy (people still run old code) and familiarity (most people who do HPC are not familiar with other kinds of languages).
Features of modern programming languages like garbage collection or run-time polymorphism are not suitable for HPC since speed matters so not sure where C# or Java or C++ come in.
That is not true in general. Classical HPC was mostly doing linear algebra with machine-precision numbers. However, modern HPC is increasingly using supercomputers for a wider variety of crunching, like symbolic calculations with arbitrary mathematical expressions instead of machine precision numbers. This places quite different characteristics on the tools you use and it is not uncommon to use programming languages other than Fortran because symbolic computation can be prohibitively difficult without GC and other kinds of optimizing compiler such as OCaml's optimizing pattern match compiler.
For example, read this paper by Fischbacher et al. which says "the authors have strong reason to believe that this may well be the largest symbolic calculation performed so far".
Basically, all the programs which do the actual work of number crunching are still FORTRAN (the old blas, lapack, arnoldi etc are still the one used)... However, when it comes to higher level structure... people are increasingly using C++.
The complexity of the simulation involve huge code and to get any kind of benefit out of writing one is to make it reusable. Also, the concepts used have also become very complex. It almost madness to represent that information using FORTRAN. That's where C++ comes in since it inherently supports Object Oriented Design. However, Run-Time Polymorphism is rarely preferred. People instead almost always use Static Polymorphism (which is implement in C++ with template meta-programming)
Also, now the compilers are really good, hence a lot of optimization is left to the compilers.
There are two kinds of problems that need to be addressed in HPC applications: one is the number crunching itself and the other is managing of computations. The first one is usually approached with code written in Fortran, C or C++ because of speed and because of the fact that there are already a lot of scientific algorithms written in this languages. The steering of computations is more conveniently implemented in higher level languages. Python is a "glue" language of choice for handling application logic and calling extensions implemented in compiled languages. Java is frequently used by projects in which managing networking and distributed computing is essential.
