Thanks for the information!
So TensorCores are used by default in PyTorch already? I did use the Nsight Systems but the results were very difficult to analyze. It could be due to the fact that TensorCores are already being used.
Just some quick follow up questions:


why does torch.backends.cudnn.benchmark = True improve speed up? Does it utilize CUDA cores in conjunction with tensor cores? I am seeing more kernels being used when monitoring with Nsight Systems. Also seems like this optimization only improves performance of AMP code, but not regular. Do you know why this is the case?


If tensor cores are used by default, are there any advantages of using AMP (other than for optimization in question 1)? because it seems like the gradient scaler already takes a lot of CPU time. I am also afraid of AMP potentially causing instabilities in network training.


Are newer version of CUDA supported by pytorch? I am using CUDA 11.1.0, which is the latest CUDA supported by pytorch according to the website.


One of the post you listed mentioned using last channel format for additional performance increase. However, when I did this with AMP, I actually observed a performance decrease. Can this occur or is something wrong with my setup?


Thanks again for the help.