By default the loss function will return a reduced loss (mean in the default setup).
You could use reduction='none', return the unreduced loss, mask or index it, calculate the loss, reduce it, and call backward() of this new loss. This approach would allow you to ignore the “border” pixels.
To do so you would need to keep the border indices around in order to be able to reuse them in the last masking step.