Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
fundamental computational unit for tensor parallelism is nn.Module. In essence, tensor parallelism consists in traversing the model and replacing specific submodules of the model with their distributed implementations. Take Aways Distributed parallel training has two high-level concepts of