Hello,
I am implementing the actor-critic algorithm. I have my actor and critic that are both neural nets but I want to train simultaneously and independently.
What I currently have looks more or less like this:
actor = Net1()
critic = Net2()
optim_a = optim.SGD(actor.parameters(),lr=0.1, momentum=0.9)
optim_c = optim.SGD(critic.parameters(),lr=0.1, momentum=0.9)
optim_a.zero_grad()
optim_c.zero_grad()
output_c = actor(input)
output_a = critic(output_c, other_inputs)
loss = output_c+output_a
loss.backward()
optim_a.step()
optim_c.step()

But when I train, the weights are not changing even though the loss is not 0, that is, there is some issue when the backpropagatiion is carried out. Furthermore, I would ideally like to have this
output_c.backward()
optim_c.step()
output_a.backward()
optim_c.step()

Because I am not sure, when I add them up, as in the first block, the optimizer modifies weights independently (I assume it does). I emphasize this because the output of the critic depends on the actor.
So, I’d appreciate any hint on how to do this or a different way to implement this.
I am aware of actor-critic pytorch tutorial 13. But they define one neural network with two independent outputs that they use to have as actor and critic, but nevertheless both share some of the weights. Ultimately, if this is the only option, then I’ll implement a shared neural network instead of two independent ones.
Thanks