I understand but the problem is when i set requires_grad=True for self.kernel the algorithm just initializes the weights based on the linear combination. Afterwards, all the coordinate of self.kernel start changing. That is not the behavior i want. I want only the coefficients to be trained. So  self.filter has to stay fixed while self.weight is changing. Therefore, it has to be the case that for self.kernel requires_grad=False.
An easy way to observe the right behavior during training is the following: coordinates 0x1, 0x2, 1x1, and 2x2 of the self.kernel has to stay zero during the entire training process. Hope that helps.
Now, my question is, is it an internal limitation of pytorch that by design you cannot define the weight of nn.conv2d or F.conv2d to be non-trainable? Do i need to change the F.conv2d function internally in pytorch?
Thank you for your help.