A fairly straightforward way (on a theoretical level, at least) of parallelizing artifical neural networks (ANNs) would be to divy up the batches of training examples during every epoch so that several workers can calculate their respective contribution to the error gradient in parallel.
This certainly helps learning when the mini-batch size is large enough, but I was wondering if there were other venues of parallelization to exploit, especially ones that would make the epoch time faster (that is, since batch-wise parallelization will take just as long per-epoch compared to a serial algorithm with stochastic gradient descent on smaller batches).
I would figure that for certain cases (such as CNNs), vectorized architectures would enable faster propagation, but I'm referring to parallelization on a higher level - for instance, if the ANN is sparsely connected then perhaps workers can run forward and backpropogation at the same time, respectively, where each worker is responsible for some densely connected component of the ANN, and use some message passing for edges along cuts.
Do any of these ideas generalize? Have ANNs been successfully parallelized, perhaps using other approaches, on this high of a level?
Apparently, something in the manner I described is implemented in SPANN. From an overview of the paper it seems like the message-passing approach is indeed appropriate for a massive ANN with many "separable" components. I would still appreciate insight in other approaches for paralleliztion, especially for smaller nets (perhaps even those that would fit onto a single machine with a lot of cores).
