Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
training efficiency and resource utilization. Taking pipeline parallelism as an example, PipeDream improves pipeline efficiency by sacrificing memory to store multiple copies of weights. TeraPipe introduces another pipelining specific to single-transformer architectures, where pipelining occurs