@richard
I have figured out an implementation error:
grad_weight += torch.mul(grad_output[i,:], weight)

should be replaced by:
grad_weight += torch.mul(grad_output[i,:], input[i,:])

and correpondingly, the second version of scripts
grad_weight = torch.sum(grad_output * input, 0)

But it dit not pass the gradient checking again.

Hadam2.png310×605 6.24 KB