ptrblck:

You could indeed build from source and compare the speed. If you are seeing a speedup it might come from a different cudnn version and/or from the usage of the missing xmma kernels in the binary.


Thank you for the suggestion!!
I built the pytorch from source following our guide 47 with cuda 11.0 and finally succeed! (P.S. I had ever tried using cuda 11.0 + conda-binaries-based pytorch, still slow, so build from source and install should be the solution.)
The running time finally comes to a satisfying level! Where the data loading took 0.038ms and training steps took only 2.05ms!
Note that at first I failed to build pytorch because a library reference not find error, as I used cuda 11.2 (though NVIDIA said the cuda 11.2 should be compatible with cuda 11.0?) So I uninstalled cuda 11.2 and installed cuda 11.0 and cudnn as normal, then successfully built pytorch from source and installed it, tested the program, whose result finally became good.
Really thank you for patiently helping me solve the problem! 
By the way, it seems that the local cuda and cudnn was indeed not used, as you said:



 ptrblck:

Your local CUDA toolkit and cudnn library won’t be used, if you’ve installed the conda binaries or the pip wheels.


But I still do not know why I can use conda-binaries-based pytorch normally on my old lab
server with RTX2080, cuda11.0?.. Maybe because the latest Ampere architecture of RTX 30xx?