I’m observing something That I’m not sure what to make of: So the output is a HxW matrix, and the loss is calculated by a selected number of values within this matrix. I found that as the training went on the unselected region would just go to zero. My network contains mostly convolutional layers,  so I’d assume the loss of the selected pixel would propagate to enough locations to be covered.
Right now I’m assuming I might not be propagating the loss through the corresponding pixel locations that are not used during the calculation of loss. Is my assumption correct? Is there a way in PyTorch to set the backprop for all values in the matrix and not just the selected ones?