I had the same problem, the model after 4 iterations took 10 times longer to inference in CPU compare to the model after 2 iterations, and I found that in the convolution layer of all residual blocks, the weights got extremely small, in my case ~ 10^-27 and as a result it took much longer time to calculate. It got too small because I trained it very long time. So I just need to revert to the previous model and train with a smaller number of epochs and it worked.