albanD:

hat if you just want to get the sum of the gradients and do a single update


Yeah now that you mention it I think I have read that before. Thank you. So to be clear, keeping the parallel path’s losses separate is not actually doing anything because autograd’s computational graph keeps track of which weights are effecting the loss. So I can then just add them up and update them based on that? Is there any scenario where I should be worried about autograd not assigning the gradient to the correct weightings that are effecting the loss? Or does it just do all of that for you?