I got this warning at the end of each epoch when using multiple GPUs:
[W CudaIPCTypes.cpp:22] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
But it doesn’t seem to affect the training since the result is as good as it is.
But I would still like to know what the probable cause is and how to solve it  
I simply use: model = torch.nn.DataParallel(model) to enable multi-GPUs.
Besides, I also added torch.multiprocessing.set_start_method('spawn', force=True) in my code. Don’t know whether it has any effect on this.
Thank you for the answer in advance.