Solved by chenglu in post #3 
              
              
                You already did what you wanted. Just sum of the different loss and then do the backward. The network will take care of everything.