I just found out about the
torch.backends.cudnn.deterministic = True

flag and gave it a try using a simple ResNet18 with MNIST. It turns out setting the flag to True actually results in a 2x speedup (~3 min instead of ~7 min). I ran the exact same code twice with and without
torch.backends.cudnn.deterministic = True

and was wondering how this is possible! I expected exactly the opposite. Can it be due to the implementation of the deterministic convolution operation, i.e., that it works just faster on smaller networks? The GPU usage is in both cases ~820 Mb.