If you are setting the parameters to zero in the module, all training will be lost.
Maybe the functional API approach would work, where you define your weight as an nn.Parameter and apply it via F.linear.
This would allow you to use a condition and either use your weight or any other dummy tensor.
Also, I’m not sure, if zeroing out the gradients in the module wouldn’t also work for the padding inputs.