That sounds like an interesting idea!



 CompRhys:

I don’t think this question will be particularly useful for other users and so feel free to delete it.


Not at all! I really like these new ideas and would like to know how your experiments worked out!
Here is a small code snippet zeroing out the activations of the linear layer based on the “missing” features of the previous input. My definition of missing is just setting the feature to zero values.
class MyDropout(nn.Module):
    def __init__(self):
        super(MyDropout, self).__init__()
        
    def forward(self, x, pre_x):
        if self.training:
            mask = (pre_x != 0).all(0).float()
            x = x * mask
        return x
        

N, C = 10, 5
x = torch.randn(N, C)
# Set a few features to zero
x[:, [0, 3]] = 0.

lin = nn.Linear(5, 5)
drop = MyDropout()

output = lin(x)
output = drop(output, x)
print(output)

Let me know, if that would work for you or if I misunderstood your use case.