Depending on the model it could be a regression in e.g. cublas, cudnn, or another library.
You could profile the code as described here or using the PyTorch profiler to try to isolate the bottlenecks.