Solved by suraj.pt in post #2 
              
              
                You can combine the losses from all heads, and backprop that using a single optimizer that is initialized with all the parameters in your model (shared network and each of the action heads). Read more about backpropagating loss in multi-task learning architectures. 
So your code might look like 
opt…