Well, I guess you can set the weights manually to zero in your forward method. Before pushing the tensor to the linear layer, you first set the weights as position padding_idx to zero. In this case, you don’t need to worry if there’s any update through backpropagation. I mean, there is, but you overwrite it each time before “using” the linear layer.