It looks like you would like to set the requires_grad flag to False for a specific kernel of your weight matrix.
This is not possible as you can only set requires_grad for the whole matrix.
However, you could zero out the gradients for this kernel after backward and before the optimizer is called.