The performance regression might come from other libraries such as cudnn and the 1.7.1 binary is using cudnn8.0.5. Could you post the model architecture and all necessary input shapes, so that we could check, which kernels are called?
Could you disable TF32 via torch.backends.cuda.matmul.allow_tf32 = False and torch.backends.cudnn.allow_tf32 = False and check the accuracy for a new run?