Thank you for answer. I changed “cuda:0” to “cuda” as you said, and didn’t see any changes. If I change “batch_size” depending on the “torch.cuda.device_count()”, I will get different sizes of batch and time comparisons will not be honest between single GPU and multi GPU.