Apart from software (pytorch, python, etc) related problems, you may also suspect hardware issues.
For instance, below is an observation that I had come across. I am not sure if this is your case too.
If you try noticing the temperature of GPU in nvidia-smi and if the temperature >= 90C, GPU might enforce software/hardware throttling that would slowdown the GPU.
In my case, for the first few loops, the temperature was less, so it was fast. However, during training due to the rise in temperature, throttle came into effect.