Thanks for the check.
Are you seeing the slowdown using the torchvision.models.densenet121 or are you changing it somehow?
I would like to reproduce it with cudnn7 as well as cudnn8 and check, why amp is slowing down the model.