Kingsmove:

The running time finally comes to a satisfying level! Where the data loading took 0.038ms and training steps took only 2.05ms!


That’s good to hear, so I assume an update in the cudnn version as well as potential calls into the missing kernels might have accelerated the workload! 



 Kingsmove:

Note that at first I failed to build pytorch because a library reference not find error, as I used cuda 11.2 (though NVIDIA said the cuda 11.2 should be compatible with cuda 11.0?)


Could you post the error message you were seeing? As we are building it with CUDA11.2, it would be good to see what wasn’t working in your setup.



 Kingsmove:

But I still do not know why I can use conda-binaries-based pytorch normally on my old lab
server with RTX2080, cuda11.0?.. Maybe because the latest Ampere architecture of RTX 30xx?


The conda binaries ship with the specified CUDA runtime (as well as cudnn, NCCL etc.), which is in your case 11.0. This CUDA release supports multiple GPU architectures, where your RTX2080 (sm_75) is included.