Solved by Kushaj in post #4 
              
              
                If I understand your problem. You only want to backpropagate through only those pixels that are True in the mask. In that case, you cannot use the approach, you defined in the question as that would backpropagate through all the pixels. 
Use this code instead 
class myModel(nn.Module):
    def __ini…