nn.DataParallel might use more memory on the default device as described in this blog post 64. We generally recommend to use DistributedDataParallel 170 to avoid these issues and to get the best performance.