Just add all loss together and call .backward() once! DO NOT respectively call ‘loss1.backward’ and ‘loss2.backward’, since you do not add ‘retain_graph =True’ in the module of ‘main_block’.
A good tutorial of ‘retain_graph=True’ is here https://github.com/pytorch/tutorials/tree/master/advanced_source 168