Thank you for the reply!
Do you meant that the backward on the shared part has been automatically taken care of by Pytorch? Maybe I should modify my question a little bit:
loss1 = alpha*loss1
loss1.backward()
loss2 =  (1-alpha)*loss2
loss2.backward()

I am pretty sure this time the shared part gets two backwards (need retrain_graph=True). Do you think there is way to somehow accumulate the gradients before the shared part and resume the backprop with the combined gradients?