in forward function, change the return code
 op1= self.branch1(X)
 op2= self.branch2(X)
 return op1,op2

and while training:
outputs = model(inputs)
loss1 = loss_fn(outputs[0],labels[:,0,:])
loss2 = loss_fn(outputs[1],labels[:,1,:]
loss = loss1+loss2
loss.backward()
opt.step()

Hope this helps!