I solved it by myself!
Because I call the self.convrelu1_backward for two times, the weights of self.convrelu1_backward is replaced when performing back-propagation. Later, I use different naming(or objects) for the two calling, and the error is solved!
self.convrelu1_backward = nn.intrinsic.qat.ConvReLU2d(32, 32,(3, 3), padding=1, bias=False, qconfig = qconfig)

        self.convrelu2_backward = nn.intrinsic.qat.ConvReLU2d(32, 32,(3, 3), padding=1, bias=False, qconfig = qconfig)


        self.conv1_backward = nn.intrinsic.qat.ConvReLU2d(32, 32,(3, 3), padding=1, bias=False, qconfig = qconfig)
        value1 = self.convrelu1_backward.weight_fake_quant # convrelu1_backward is fused with relu, but I only want the conv (without relu), and take out the weight
        self.conv1_backward.weight_fake_quant = copy.deepcopy(value1)
        self.conv2_backward =nn.qat.Conv2d(32, 32,(3, 3), padding=1, bias=False, qconfig = qconfig)
        value2 = self.convrelu2_backward.weight_fake_quant # convrelu2_backward is fused with relu, but I only want the conv (without relu), and take out the weight
        self.conv2_backward.weight_fake_quant = copy.deepcopy(value2)