Are you using the same PyTorch installation (conda binaries, pip wheels, or a source build) on both machines?
If so, are you also using the same CUDA and cudnn version? In case you are using cudnn (e.g. in conv layers), are you using torch.backends.cudnn.benchmark = True?