Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
like GPT-3 and DALL-E 2. There are two primary types of distributed parallel training: data parallelism and model parallelism. We further divide the latter into two subtypes: pipeline parallelism and tensor parallelism. We will cover all distributed parallel training here and demonstrate how to