Could you please post the slower layers here?
To isolate the issue further, I would also recommend to use the same CUDA and cudnn versions for the different PyTorch versions. Otherwise you are facing a lot of variables, which might affect the performance.