Did anyone find what was going on here. I am having maybe the same problem. Early epochs of training take ~27 s, while after 40 epochs times can be up to ~150 s.
The speed (and change thereof) for me is also quite dependent upon the learning rate. Also I have mostly been using the optimiser Adam, but I also get it with SGD.
I have looked quite a bit into what effects this, and roughly where the slow is coming in, so could post more if there are people who are curious.