I've spent the last few days working with tensorflow for the first time as part of a natural language processing assignment for my degree. It's been interesting (fun isn't the right word) trying to get it to run on a GPU but it got me thinking.
The recent advances in deep learning have come about as GPGPU technologies have matured and the frameworks have arrived to make doing massive amounts of linear algebra on your computer much quicker and easier. Nvidia now sell chips that are designed specifically for this task and from what I understand, papers like the one featuring AlexNet would not have been possible without GPU acceleration. This point is nicely articulated by the authors:
All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.
Given this, my question is then why haven't we seen more adoption of GPUs for traditional HPC tasks (simulation, rendering etc.)? These workloads have been around for years yet it seems like only recently that GPGPU has taken off as an approach. It appears to me that the requirements are pretty similar, namely 'read a load of data, do a load of floating point transforms on the data, save some data, repeat' but a look at the TOP500 reveals that many of the systems on there are still using CPUs (although an increasing number using 'manycore' processors like the Intel Phi which seem to straddle the CPU/GPU divide).
Are there actually less similarities between traditional HPC and large-scale ML workloads than I imagine? Maybe it's that GPUs are less efficient in terms of flops/w which is what really matters when it comes to running a huge compute cluster? Is it just down to a very effective marketing effort by Nvidia in the machine learning space?