Answers.Body: <p>Instead of receiving total return in the summation form of (discounted) rewards sampled from the full trajectory as in REINFORCE with baseline to update policy parameters, in actor-critic method to bootstrap, the actor sub-network usually receives a one-step return in the familiar form of temporal difference (TD) error <span class="math-container">$(R_{t+1} + \gamma \hat V_{\pi_\theta}(S_{t+1}, w) - \hat V_{\pi_\theta}(S_t,w))$</span> (or n-step) from the critic who evaluates estimated state value function optimized from same TD error.</p>
  <p>Obviously from the computational point of view the policy gradient's &quot;advantage&quot; function from the critic's one-step or n-step TD error has much lower variance than that of REINFORCE with baseline from its sampled trajectory's total return, even if the former introduces bias.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-12T07:17:12.770Z
Answers.LastActivityDate: 2022-12-12T07:22:38.497Z
Answers.LastEditDate: 2022-12-12T07:22:38.497Z
Answers.UserDisplayName: cinch
Body: <p>I understand the general idea behind the Actor-Critic architecture. The actor maps state to action, and the critic maps state + action to reward.</p>
  <p>But I don't fully understand how the critic output (reward space) can be used to train the actor sub-network.</p>
  <p>How can a reward be used to train the actor sub-network to make it choose a better action when the actor sub-network doesn't get the reward as input or output?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-21T05:02:05.033Z
Comments.Text: As shown in my below answer, the only difference between actor-critic method and REINFORCE with baseline is the former also applies the same state value estimator to the *next* state of any state transition to form the TD error and thus is a kind way of assessing the latest done action before the state transition to further update actor network parameters, while the latter's state value estimator doesn't assess the action taken at any same iteration during an episode via any of the next state value at all to influence its next action.
Comments.UserDisplayName: cinch
Tags: <reinforcement-learning><machine-learning><deep-learning><deep-rl><actor-critic-methods>
Title: How the Critic is used to train the Actor in Actor-Critic network
UserDisplayName: nrofis
ThreadText: # How the Critic is used to train the Actor in Actor-Critic network
  
  (Asked by: username_0 on 2022-12-09)
  
  I understand the general idea behind the Actor-Critic architecture. The actor maps state to action, and the critic maps state + action to reward.
  
  But I don't fully understand how the critic output (reward space) can be used to train the actor sub-network.
  
  How can a reward be used to train the actor sub-network to make it choose a better action when the actor sub-network doesn't get the reward as input or output?
  
  **Tags:** reinforcement-learning, machine-learning, deep-learning, deep-rl, actor-critic-methods
  > **Comments:**
  > **username_1**: As shown in my below answer, the only difference between actor-critic method and REINFORCE with baseline is the former also applies the same state value estimator to the *next* state of any state transition to form the TD error and thus is a kind way of assessing the latest done action before the state transition to further update actor network parameters, while the latter's state value estimator doesn't assess the action taken at any same iteration during an episode via any of the next state value at all to influence its next action.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-12-12. Score: 1
  
  Instead of receiving total return in the summation form of (discounted) rewards sampled from the full trajectory as in REINFORCE with baseline to update policy parameters, in actor-critic method to bootstrap, the actor sub-network usually receives a one-step return in the familiar form of temporal difference (TD) error $(R\_{t+1} + \gamma \hat V\_{\pi\_\theta}(S\_{t+1}, w) - \hat V\_{\pi\_\theta}(S\_t,w))$ (or n-step) from the critic who evaluates estimated state value function optimized from same TD error.
  
  Obviously from the computational point of view the policy gradient's "advantage" function from the critic's one-step or n-step TD error has much lower variance than that of REINFORCE with baseline from its sampled trajectory's total return, even if the former introduces bias.
  
  
  ---
  
  