Hi,
If you are using custom network/loss function, it is also possible that the computation gets more expensive as you get closer to the optimal solution?
To track this down, you could get timings for different parts separately: data loading, network forward, loss computation, backward pass and parameter update. Hopefully just one will increase and you will be able to see better what is going on.