Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
(1/11/2021), the Beijing Academy of Artificial Intelligence (BAAI) released the initial Wu Dao 1.0. Before long, Wu Dao 2.0 debuted on 5/31/2021 as the largest language model with 1.75 trillion parameters and ten times GPT-3 parameters. Suppose we train GPT-3 on 240 ml.p4d.24xlarge instances of the