Machine Learning, Mpi, Pytorch, TensorFlow.
For large-scale training of neural networks you need to specify how the work is broken up across machines. This is where communication primitives come in play. There are three common approaches used in modern frameworks: MPI-collectives, task-based and computation-graph based. MPI-collectives