You can combine the losses from all heads, and backprop that using a single optimizer that is initialized with all the parameters in your model (shared network and each of the action heads). Read more about backpropagating loss in multi-task learning architectures 4.
So your code might look like
optim = optimizer(full_model.parameters(), ...)
combined_loss = weighted_sum(loss1, ... lossN)
optim.zero_grad()
combined_loss.backward()
optim.step()

Action_head1 will be updated by gradients only w.r.t. loss1 (and so on), and the shared network will be updated by gradients wrt to all losses.