Ditlev_Jorgensen:

However after applying optimization


That might indicate that your learning rate is too high or your loss function is bad.
Once the softmax saturates, backprop will have a problem.
Best regards
Thomas