Solved by ptrblck in post #2 
              
              
                You could add torch.backends.cudnn.benchmark = True to let cuDNN profile different kernels and select the fastest one and use to(memory_format=torch.channels_last) since you are using amp, which could give another speedup if permutations are avoided.