I’ve also ran into problems like this, have you tried to use nn.DistributedDataParallel instead? It appears to be recommended to use that instead of nn.DataParallel even for multi-gpu on a single node:
DataParallel — PyTorch 1.9.0 documentation 13

image768×144 12.2 KB

CUDA semantics — PyTorch 1.9.0 documentation 4

image827×385 42.5 KB