Solved by Lening_Li in post #12 
              
              
                Hi, I think I have found the problem. After I use the backward, I use the network’s forward to get some other values. 

Which may cause the weights to change (I am not sure). Basically, my steps are the following: 

backwork, updates the weights
forward, may change the weights (not sure) 
Is there a…