Machine Learning, Mpi, Pytorch, TensorFlow.
HogWild, localSGD, Asynchronous Decentralized Parallel Stochastic Gradient Descent, can be implemented naturally using the primitives. A few examples are shown in our paper “Flexible Primitives for Distributed Deep Learning.” TensorFlow computation graph is somewhere in between — theoretically you