I’m not sure I’m understanding your issue correctly.
You could just use the standard loss function using the average of your batch loss, sum the losses together, and call backward on the sum of losses.