@ptrblck
Every input shape is a new one 
Is there a way we can explicitly tell cudnn to select a faster kernel instead of the bad one?
Have you tried this script on cudnn8 without setting  torch.backends.cudnn.benchmark = True?
By the way, do you know what does the kernel cudnn::detail::dgrad2d_alg1_1 do? I searched Google but couldn’t get any useful info.