Thanks for your reply! Actually, I’d like to update the model with numerous parameters, but found it consuming a lot of time in copying model across GPUs when using nn.DataParallel(). So, I’m wondering if it is possible to update the parameters in CPU using multiprocessing (GPUs are used in parallel to calculate grads). But I found it very time consuming as well. Thanks to torch.distributed, the training process has been accelerated even up to 4 times comparing with nn.DataParallel() (because torch.distributed do not copy the whole parameters across GPUs and reallocate the model in each iter).