Solved by ptrblck in post #4 
              
              
                I’m not familiar with TensorFlow, but it seems their “auto tune” functionality corresponds to torch.backends.cudnn.benchmark = True mode.