This is an interesting one that I’ve seen before in similar situation, what’s going on is a little complex, and there is nothing you can do about it.
I’m going to guess that you’re either running this on the GPU? Well, it turns out that reductions on the GPU (like sum for matrix multiplication and softmax) are actually slightly non-deterministic, in that the reduce operation will not be applied in the same order. Most reduction operations are commutative (they can be applied in any order and still get the same results), however because we are using floats (I’m guessing you’re using 32 bit?) we have a slightly different rounded value because we calculated in different order.
Initially this difference in rounding is only slight, and doesn’t show up in your output at all (if you print out more decimal places you should see the effect earlier), but it accelerates as the fixed-precision errors accumulate, and eventually it will result in such significantly different values that it takes your optimisation off in a completely different direction, and we get significant divergences as above.
There is nothing you can do about this if you are after repeatable results on the GPU, GPU computation is inherently stochastic. However you can try:

running your experiments repeatedly (with different weight initialisations as well) and average them to get a more robust estimate of your score
run your experiments for a shorter time (until before the divergence occurs)
run your experiments on a single core CPU