Didn’t go through your code, but in general a possible cause for this type behaviour could be a feature set that isn’t normalised properly.
For example one of the features isn’t normalised at all and has for example values between 0 and 255. This  causes the changes to the weights by the optimiser to overshoot when applied to this feature and it is not going in a more straight line to the optimum (hence the zigzag).
I assume at around 8000 iterations you reduce the learning rate and the changes to the weights are reduced and so is the “overshoot” factor (so smaller zigzag).