Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
architecture to scale out. In addition to the two basic types of parallelism, there are many more variants, such as expert parallelism. Furthermore, they can be mixed with two or all, such as data and model mixed parallelism. It’s common to mix both model and data parallelism for large-scale