I’ve seen similar behavior in the past, where a “bad update step” produced large gradients and pushed the model parameters out of the “nice” loss landscape.
I think lowering the learning rate helped, but I would also assume that e.g. Adam should reduce this issue.
Anyway, you could check the gradients for the update steps and compare their magnitude, in case you want to debug your model.