Thanks for your solution! I’ve got a similar problem I believe. I’m approximating a 2D function on a grid such that my output is of size Nx2xHxW.
I have training data for each of the N samples in the form of 2xHxW matrix, however, it has some of the 1x2 tuples missing from the HxW grid, so I set those values to an unacceptable value, say 999, before computing the loss. I compute an MSE loss with a mask similar to yours and exclude those output nodes from adding to the loss.
Input to the network is of size Nx2 and I have a decoder architecture with a linear layer followed by M ConvTranspose2d layers.
My questions are:

Do you agree that I should not backpropagate from all the output nodes for which I have the missing training data?
Secondly, should I dropout all the nodes from all the layers that correspond to the output nodes with missing training data?
Third, how do I achieve it for say a simple network with 1 linear layer followed by BatchNorm and Activation, 1 ConvTranspose2d layer followed by BatchNorm and Activation and 1 Output ConvTranspose2d layer?

Please help as my network is not learning with missing data.