Masking pixels for a CNN I'm trying to implement a CNN with RGB and depth images. But my depth images are a little sparse. So I would like to mask out those neighborhoods in the RGB image where the depth neighbors are empty.
I would like to mask the input feature map, i.e. wherever the neighbor pixels are empty, multiply this with zeros. While I have not yet thought out the specifics of this operation, I have a broader question.
Is masking a "safe" operation? Can I mask out random portions of my input? I believe that this is similar to dropout, but dropout has its own backprop operations.
Does it make sense to mask out input feature map, without writing a custom backward operation? If so, what are some specifics I need to keep in mind?
My ultimate goal is just to ignore the portions that have no corresponding depth value.
Thanks
Edit: Please do not suggest State of the work art on RGB+Depth fusion, that combine the two in a fancy way. I'm purely just interested in the effects of masking

The operation of putting a value to zero, removes the gradients (see the ReLU activation function). You might want to consider that by putting a large part of your image to zero, you are giving the neural network a quite skewed input distribution.
I would normally put this down as a comment, but I dont have enough reputation.
