Thanks for the information!
The error after 200 samples is approx 1e-7, while float32 precision is ~1e-6, so that could be the reason.
Just for the sake of debugging, you could use float64 and see, if the differences occur a bit later in the training.