Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
Amazon SageMaker training platform, the whole model will take 25 days to train. The challenge is not just processing but also memory. Wu Tao 2.0 appears to need more than 1000 GPUs only to store its parameters. It is imperative to employ distributed parallel training for deep learning large models