Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
illustrates how model weights are divided across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g., larger dff sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core