I have some code that uses multiple loss functions and therefore calls loss.backward(retain_graph=True) multiple times.
I heard that doing only a single backward pass can be more efficient (time-wise). Should I in that case sum all the losses or average them?
And is there any difference between calling backward on each loss separately aggregating them?