Machine Learning, Mpi, Pytorch, TensorFlow.
segmented ring, is what’s typically used for large-scale stochastic gradient descent. You can introduce more complicated aggregation schemes by making use of “rank” environment variable or running aggregation on a selected group of processes. HPC community has been refining these primitives for a