The slowdown might have been introduced by non-trained cudnn heuristics for the 3080, e.g. in cudnn8.0.5. Based on your setup description I assume you’ve built PyTorch from source, since you are using CUDA11.2 while the 1.7.1 binaries ship with CUDA11.0? If that’s the case, which cudnn version have you used to build it?
Could you also post your model, so that we could check for regressions in internal cudnn builds?