Yes, profiling would be useful, to see which part of the code slows down.
Just to clarify the issue, do you see a sudden jump in epoch 2, iteration 300 or is the time increasing steadily after a certain iteration?
If the system isn’t a bottleneck due to overheating etc., you would often see a slowdown, if you are increasing the computation graph (e.g. if you are not detaching some states in an RNN), but this should also increase the device memory usage.