I trying to implement B-CNN: Branch Convolutional Neural Network for Hierarchical Classification.-https://arxiv.org/pdf/1709.09890.pdf 13

I have a doubt-
Should I backpropagate losses from the branches individually (using retain graph)
OR
Should i simply add the losses from all the branches and then perform backpropagation.