Solved by TYC in post #2 
              
              
                I solved it by myself! 
Because I call the self.convrelu1_backward for two times, the weights of self.convrelu1_backward is replaced when performing back-propagation. Later, I use different naming(or objects) for the two calling, and the error is solved! 
self.convrelu1_backward = nn.intrinsic.qat.C…