Answers.Body: <p>The same argument of a state-value function int <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> Section 13.5 can be applied to state-action values. The main takeaway is that a critic's state value (or action-state value) function is used for bootstrapping.</p>
  
  <blockquote>
    <p>Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not consider it to be an actor–critic method because its state-value function is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating the value estimate for a state from the estimated values of subsequent states), but only as a baseline for the state whose estimate is being updated. This is a useful distinction, for only through bootstrapping do we introduce bias and an asymptotic dependence on the quality of the function approximation. As we have seen, the bias introduced through bootstrapping and reliance on the state representation is often beneficial because it reduces variance and accelerates learning.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-22T20:56:38.747Z
Answers.Text: Interesting, so do you mean that the pytorch implementation of actor-critic (https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py) is inaptly named and should rather sound something in the line of "REINFORCE with function approximator as baseline"? As I understand that is the case because we adjust the advantage based on the true returns (not bootstrapped)
Answers.UserDisplayName: d56
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-25T17:14:27.773Z
Answers.LastActivityDate: 2019-01-25T17:14:27.773Z
Answers.UserDisplayName: Jaden Travnik
Body: <p>Following <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py" rel="nofollow noreferrer">Pytorch's actor critic</a>, I understand that the critic is a function mapping from the <strong>state</strong> space to the reward space, meaning, the critic approximates the <strong>state-value funcion</strong>.</p>
  
  <p>However, according to <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwia87rurYngAhUH36QKHd5QDCIQFjAAegQIChAC&amp;url=https%3A%2F%2Farxiv.org%2Fpdf%2F1512.07679&amp;usg=AOvVaw23yfpdaGmo9gs3_AIz8edQ" rel="nofollow noreferrer">This paper</a> (you don't need to read it, just a glance at the nice picture at page 2 is enough), the critic is a function mapping from the <strong>action</strong> space to the reward, meaning it approximates the <strong>action value funcion</strong></p>
  
  <p>I am confused.</p>
  
  <p>When people say "actor critic" - what do they mean by "critic"?</p>
  
  <p>Is the term "Critic" ambiguous in RL?</p>
  
Tags: <machine-learning><reinforcement-learning><open-ai>
Title: What does the Critic network evaluate in Actor Critic?
UserDisplayName: Gulzar
ThreadText: # What does the Critic network evaluate in Actor Critic?
  
  (Asked by: username_0 on 2019-01-25)
  
  Following [Pytorch's actor critic](https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py), I understand that the critic is a function mapping from the **state** space to the reward space, meaning, the critic approximates the **state-value funcion**.
  
  However, according to [This paper](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=2ahUKEwia87rurYngAhUH36QKHd5QDCIQFjAAegQIChAC&url=https%3A%2F%2Farxiv.org%2Fpdf%2F1512.07679&usg=AOvVaw23yfpdaGmo9gs3_AIz8edQ) (you don't need to read it, just a glance at the nice picture at page 2 is enough), the critic is a function mapping from the **action** space to the reward, meaning it approximates the **action value funcion**
  
  I am confused.
  
  When people say "actor critic" - what do they mean by "critic"?
  
  Is the term "Critic" ambiguous in RL?
  
  **Tags:** machine-learning, reinforcement-learning, open-ai
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-25. Score: 2
  
  The same argument of a state-value function int [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/bookdraft2017nov5.pdf) Section 13.5 can be applied to state-action values. The main takeaway is that a critic's state value (or action-state value) function is used for bootstrapping.
  
  > Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not consider it to be an actor–critic method because its state-value function is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating the value estimate for a state from the estimated values of subsequent states), but only as a baseline for the state whose estimate is being updated. This is a useful distinction, for only through bootstrapping do we introduce bias and an asymptotic dependence on the quality of the function approximation. As we have seen, the bias introduced through bootstrapping and reliance on the state representation is often beneficial because it reduces variance and accelerates learning.
  
  > **Comments:**
  > **username_2**: Interesting, so do you mean that the pytorch implementation of actor-critic (https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py) is inaptly named and should rather sound something in the line of "REINFORCE with function approximator as baseline"? As I understand that is the case because we adjust the advantage based on the true returns (not bootstrapped)  
  
  ---
  
  