I believe my proposal does just that.
residual is calculated as the mse of a single pixel, and loss is calculated from the residual of that single pixel (I assume bs is the batch size), so when you do loss.backward() it will backpropagate through the calculations that lead to that pixel.