Oh, I’m sorry Tom, it’s my fault:tired_face:. It is not my invention, in fact, I am trying to implement the “XNOR network” on PyTorch. The inventor of that network wants to make the network, especially the conv layers in the network to be smaller to store and faster to forward propagate.
His method is to set another “weight”, a binarized one W’ , in each conv layer. That is, in each conv layer, there are two weight, a normal one we are familiar with in other the state-of-the-art network W, and W’:

W’ = sign(W)*mean(abs(W))

When doing forward propagate, the author replace

conv_output = activation_function(conv_input*W)

By his new thought:

W’ = sign(W)* mean(abs(W))
conv_output = activation_function(conv(conv_input,*W’))

While in backward propagation, the gradient of W would be NAN because of some indifferentiable function like “sign()”. So the author use the grad of W’ to update W like:

W := W - learning_rate * W’.grad()

And that is the point I am stuck. I could have the “W’.grad()” computed and (it seems that) the gradient is right, while I don’t know how to update W. I have tried to use:

self.weight = Parameter(torch.Tensor(
out_channels, in_channels // groups, *kernel_size), requires_grad=True).cuda()
self.Bi_weight = Parameter(torch.Tensor(self.weight.shape))


    for layer in model.modules():
        if isinstance(layer, nn.Conv2d_Bi):
            layer.weight.grad = copy.deepcopy(layer.Bi_weight.grad)


and “optimizer.step()” but failed. I am trying something else. Do you have any advice or thought?
Thank you Tom~:stuck_out_tongue_closed_eyes: