Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
References Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training: https://arxiv.org/abs/2111.05972 Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity: https://arxiv.org/abs/2101.03961 Language Models are Few-Shot