Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
Transformer-based language models have been stealing the show. The notorious GPT-3 blew out with 175 billion parameters and 96 attention layers with a 3.2 M batch size and 499 billion words. Exactly half a year later, Google published Switch Transformer with 1.6 trillion parameters. On the same day