Answers.Body: <p>I am also very curious about this. I have been implementing A2C in PyTorch from scratch and have tried both a single optimizer and separate optimizers. The separate case learns so much quicker. I believe it may have something to do with the coefficients, balancing the critic coefficient so the two losses were within a similar range seemed to work.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-26T14:36:33.953Z
Answers.LastActivityDate: 2022-02-26T14:36:33.953Z
Answers.UserDisplayName: Elfurd
Body: <p>Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we can define 2 separate optimizers for the Actor and the Critic and the backward process will be</p>
  <pre><code>actor_loss.backward()
  actor_optimizer.step()
  critic_loss.backward()
  critic_optimizer.step()
  </code></pre>
  <p>or we can use a single optimizer for both the Actor's and the Critic's parameters so the backward process can be like</p>
  <pre><code>loss = actor_loss + critic_loss
  loss.backward()
  optimizer.step()
  </code></pre>
  <p>I have 2 questions regarding both approaches:</p>
  <ol>
  <li><p>Is there any consideration (pros, cons) for both the single joined optimizer and the separate optimizer approach?</p>
  </li>
  <li><p>If I want to save the best Agent (Actor and Critic) periodically (based on a predefined testing environment), do I always have to update the Critic, regardless of the current Agent's performance? Because (CMIIW) the Critic is (in its most basic purpose) only for predicting the action-value or state-value thus a more trained Critic is better.</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-17T02:38:19.243Z
Comments.Text: You're asking 2 distinct questions here. I would suggest that you ask the second question in a separate post.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><pytorch><actor-critic-methods><optimizers>
Title: Joined vs Separate optimizer for Actor-Critic
UserDisplayName: Sanyou
ThreadText: # Joined vs Separate optimizer for Actor-Critic
  
  (Asked by: username_0 on 2021-09-11)
  
  Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we can define 2 separate optimizers for the Actor and the Critic and the backward process will be
  actor_loss.backward()
  actor_optimizer.step()
  critic_loss.backward()
  critic_optimizer.step()
  
  or we can use a single optimizer for both the Actor's and the Critic's parameters so the backward process can be like
  loss = actor_loss + critic_loss
  loss.backward()
  optimizer.step()
  
  I have 2 questions regarding both approaches:
  
  Is there any consideration (pros, cons) for both the single joined optimizer and the separate optimizer approach?
  
  If I want to save the best Agent (Actor and Critic) periodically (based on a predefined testing environment), do I always have to update the Critic, regardless of the current Agent's performance? Because (CMIIW) the Critic is (in its most basic purpose) only for predicting the action-value or state-value thus a more trained Critic is better.
  
  **Tags:** reinforcement-learning, deep-rl, pytorch, actor-critic-methods, optimizers
  > **Comments:**
  > **username_1**: You're asking 2 distinct questions here. I would suggest that you ask the second question in a separate post.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-26. Score: 1
  
  I am also very curious about this. I have been implementing A2C in PyTorch from scratch and have tried both a single optimizer and separate optimizers. The separate case learns so much quicker. I believe it may have something to do with the coefficients, balancing the critic coefficient so the two losses were within a similar range seemed to work.
  
  
  ---
  
  