Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
parallelism and distribution. Parallelism is a framework strategy, and distribution is an infrastructure architecture. Distributed parallel training is critical but still nascent in the industry and research. We can look forward to three creative areas happening in the future. Parallelism shards