Answers.Body: <p>TRPO still uses the <strong>advantage function</strong>, which computed using the critic (aka value function).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-26T02:53:46.273Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-25T21:25:31.087Z
Answers.LastActivityDate: 2023-01-21T17:56:49.873Z
Answers.LastEditDate: 2023-01-21T17:56:49.873Z
Answers.UserDisplayName: lejlot
Body: <p>From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size <span class="math-container">$\beta$</span> from a KL constraint between the new and old policy.</p>
  <p>NPG is a modification to &quot;vanilla&quot; policy gradient algorithms that modifies the direction of the update using the Fisher information matrix such that the direction is in that of maximal improvement per unit of KL divergence between new and old policies.</p>
  <p>Now, I've skipped a lot, but throughout most introductory material on these two algorithms, I see no mention of a critic (i.e. auxiliary evaluator value function), yet  know that at least TRPO is considered actor-critic. Why is that? Where does the critic come in?</p>
  
Tags: <reinforcement-learning><policy-gradients><actor-critic-methods><trust-region-policy-optimization><policy-based-methods>
Title: What makes TRPO an actor-critic method? Where is the critic?
UserDisplayName: thesofakillers
ThreadText: # What makes TRPO an actor-critic method? Where is the critic?
  
  (Asked by: username_0 on 2022-10-25)
  
  From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and old policy.
  
  NPG is a modification to "vanilla" policy gradient algorithms that modifies the direction of the update using the Fisher information matrix such that the direction is in that of maximal improvement per unit of KL divergence between new and old policies.
  
  Now, I've skipped a lot, but throughout most introductory material on these two algorithms, I see no mention of a critic (i.e. auxiliary evaluator value function), yet know that at least TRPO is considered actor-critic. Why is that? Where does the critic come in?
  
  **Tags:** reinforcement-learning, policy-gradients, actor-critic-methods, trust-region-policy-optimization, policy-based-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-25. Score: 2
  
  TRPO still uses the **advantage function**, which computed using the critic (aka value function).
  
  > **Comments:**
  > **username_2**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  