Thanks for the reply again 
I change the momentum to zero but it still changes the gradient and the weight still updates.
So I made the weight decay to zero and it works! I wonder why weight decay makes this difference.
P.S If i have to set weight decay to zero, then is there other solution to give regularization like the weight decay do?