I’d like to add the ability to ignore padding indexes to nn.Linear, what is the best way to manually set the weights/bias of an nn.Linear to 0 when a certain value is passed into the layer from the input features?
In my specific application, I have an embedding for protein sequences that maps numbers (tokenized amino acids) to vectors, but I need to map those vectors to the size of my model. It seems natural to use nn.Linear to receive the tensor of shape (sequence, embedding_dimension) and output to shape (sequence, model_dimension), but I need to ignore padding. Specifically, the embedded padding token would be this vector (-5, -5, -5,-5, -5, -5,-5, -5, -5) (the embedding dimension is 9), so I want to set the weights/bias of nn.Linear to 0 whenever it encounters a -5.