Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
general and flexible framework, supporting pipeline and tensor parallelism with memory-saving features. Its pipeline parallelism engine enables load-balancing auto-partitioning and pipelining runtime for arbitrary model architectures based on module-server design. As with pipeline parallelism, the