Hi,
I am not a specialist but the usual reasons I have seen are:

learning rate too high
loss function is non-smooth for low values

This is most likely the second one here if it behaved properly at the beginning of the training.
You want to make sure that small loss value don’t make anything numerically unstable.