hi Pan-Zhou,
I am not sure of exactly why you are seeing this behavior. If you pin it down, I would love to know.
Some easy things to try:

increase / decrease learning rate, see what happens
print out the min/max values of the weights of the network over learning, as well as the norm.
check if somehow weights are becoming NaN.