If you know the position of the irrelevant pixels,  you could create a mask with zeros for them (and ones for the valid locations), and multiply your unreduced loss with it (reduction='none' for the loss function initialization). Afterwards you could calculate the mean (or normalize the loss somehow) and call .backward() as usual.