Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
Horovod mainly accomplishes the following three tasks: First, it creates and dispatches copies of the model, one copy per each accelerator. It shards the data and then distributes it to the corresponding devices. It finally aggregates all results together in the backpropagation step. So we can see