You’re right, I didn’t notice that there are additional learnable parameters used in the quantization which need gradients.
Then I think the most clean way to do so is to just customize your own Conv2d class, which in the forward function just takes the quantized weights as the argument of F.conv2d.