This effect is often observed when decreasing the learning rate, as your loss might be “stuck” due to too large gradient steps. E.g. the ResNet paper 6 shows the same behavior in the loss curves (this paper of course doesn’t discuss this effect).