I’ve had a different problem whereby if I trained with PyTorch on GPU, it was suboptimal compared to training the same thing on PyTorch with CPU. The problem was the CUDNN libraries used with GPU. Thus using torch.backends.cudnn.enabled = False solved the problem for me. Maybe it solves your problem too?