I found that when using an architecture that is more rare, it is really important to set torch.backends.cudnn.benchmark = True
This made training run 5x faster in my older Maxwell Titan and it could also apply to the Turing cards.