Ok, so in this configuration, the mask is applied only on evaluation? Because when I do model(X_test) the mask seems to be applied.
I tried what you suggested, namely:
class DropFeatures(nn.Module):
    def __init__(self, input_dim, *kargs, **kwargs):
        super(DropFeatures, self).__init__()
        self.weights = nn.Parameter(new_mask)
        
    def forward(self, x):
        return (x * self.weights)

The problem is that if I train the model setting new_mask = torch.zeros(1, 69) the loss is actually decreasing. Instead should not as all the features are zeroed, correct me if I am wrong.