Do you resume the training with a small enough learning rate? The loss can easily increase if we are using the default learning rate value, because it’s very likely to overshoot the current local minima.