Answers.Body: <p>Indeed in the sense of benchmarking the baseline function in both methods simply acts as a usual baseline to assess the <span class="math-container">$\delta$</span> as you highlighted which obviously seems to influence the policy. But the thing is in actor-critic method the <span class="math-container">$\mathbf{w}$</span>-parameterized state-value network really acts as a critic to estimate both the first (baseline) <em>and</em> the second state value functions to form the TD(0) error to influence the actor's policy. While in REINFORCE with baseline method, the critic is essentially the actual sampled return as clearly proved by the book using the fact that the baseline function doesn't depend on any action <span class="math-container">$a$</span> taken during the corresponding update step. Thus here the baseline function isn't a critic at all.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-21T01:57:30.803Z
Answers.LastActivityDate: 2024-07-21T01:57:30.803Z
Answers.UserDisplayName: cinch
Body: <p>As stated in Sutton and Barto:</p>
  <blockquote>
  <p>In REINFORCE with baseline, the learned state-value function estimates the value of
  the first state of each state transition. This estimate sets a baseline for the subsequent
  return, but is made prior to the transition’s action and thus cannot be used to assess that
  action.</p>
  </blockquote>
  <p>I have doubt regarding the use of baseline function. In actor-critic methods, a one-step return is used rather than the Monte-Carlo return to estimate the reward and is given by:</p>
  <p><a href="https://i.sstatic.net/bDvG8PUr.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bDvG8PUr.png" alt="enter image description here" /></a></p>
  <p>And as the policy parameters are nudged in the direction of more return, the baseline function here is dictating the policy. But what about the baseline term in REINFORCE Algorithm?</p>
  <p><a href="https://i.sstatic.net/F0e9zPUV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/F0e9zPUV.png" alt="enter image description here" /></a></p>
  <p>Aren't the policy parameters being changed in the direction of <code>G-v(S_t, w)</code> here and thus <code>v(S_t)</code> is still influencing our policy ?</p>
  
Tags: <reinforcement-learning><actor-critic-methods><reinforce>
Title: Doubt regarding Actor-Critic method
UserDisplayName: DeadAsDuck
ThreadText: # Doubt regarding Actor-Critic method
  
  (Asked by: username_0 on 2024-07-20)
  
  As stated in Sutton and Barto:
  
  > In REINFORCE with baseline, the learned state-value function estimates the value of
  > the first state of each state transition. This estimate sets a baseline for the subsequent
  > return, but is made prior to the transition’s action and thus cannot be used to assess that
  > action.
  
  I have doubt regarding the use of baseline function. In actor-critic methods, a one-step return is used rather than the Monte-Carlo return to estimate the reward and is given by:
  
  [![enter image description here](https://i.sstatic.net/bDvG8PUr.png)](https://i.sstatic.net/bDvG8PUr.png)
  
  And as the policy parameters are nudged in the direction of more return, the baseline function here is dictating the policy. But what about the baseline term in REINFORCE Algorithm?
  
  [![enter image description here](https://i.sstatic.net/F0e9zPUV.png)](https://i.sstatic.net/F0e9zPUV.png)
  
  Aren't the policy parameters being changed in the direction of `G-v(S_t, w)` here and thus `v(S_t)` is still influencing our policy ?
  
  **Tags:** reinforcement-learning, actor-critic-methods, reinforce
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-21. Score: 1 | Accepted answer
  
  Indeed in the sense of benchmarking the baseline function in both methods simply acts as a usual baseline to assess the $\delta$ as you highlighted which obviously seems to influence the policy. But the thing is in actor-critic method the $\mathbf{w}$-parameterized state-value network really acts as a critic to estimate both the first (baseline) *and* the second state value functions to form the TD(0) error to influence the actor's policy. While in REINFORCE with baseline method, the critic is essentially the actual sampled return as clearly proved by the book using the fact that the baseline function doesn't depend on any action $a$ taken during the corresponding update step. Thus here the baseline function isn't a critic at all.
  
  
  ---
  
  