ptrblck:

The performance regression might come from other libraries such as cudnn and the 1.7.1 binary is using cudnn8.0.5. Could you post the model architecture and all necessary input shapes, so that we could check, which kernels are called?
Could you disable TF32 via torch.backends.cuda.matmul.allow_tf32 = False and torch.backends.cudnn.allow_tf32 = False and check the accuracy for a new run?


Disabling TF32 seems to be the solution since using FP16 training in my task will also brings performance regression. I will further investigate on this issue and findings will updated below once finished.
Thanks a lot!