Body: <p>I am reading about the actor-critic architecture. I am confused about how the actor determines the action using the value (or future reward) from the critic network.</p>
  
  <p>Below you have the most popular picture of actor-critic network. </p>
  
  <p><a href="https://i.sstatic.net/pKKmS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pKKmS.png" alt="Actor Critic Network"></a></p>
  
  <p>It looks like the input of the actor network is only the "state" variable (<span class="math-container">$s_t$</span>), it has nothing to do with the critic network.</p>
  
  <p>However, from the equation below</p>
  
  <p><a href="https://i.sstatic.net/OKy4l.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OKy4l.png" alt="Policy Gradient with Critic Network"></a></p>
  
  <p>the actor seems to be related to critic network. </p>
  
  <p>I have a few questions</p>
  
  <ol>
  <li><p>Does actor network has two inputs, state variable and future reward (output from critic network), or only the state variable?</p></li>
  <li><p>If the actor network does take future reward as input, how does it use it? Only during the training stage, or in the action making stage?</p></li>
  <li><p>Is there a "policy iteration" procedure happens during decision making stage, i.e. for every state <span class="math-container">$s_t$</span>, policy network will make several attempts with critic network, and output the best policy? </p></li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-17T21:00:18.933Z
Comments.Text: "It looks like the input of the actor network is only the "state" variable (), it has nothing to do with the critic network.". Note that the picture contains an arrow from the critic which crosses the the actor.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-17T21:04:36.630Z
Comments.Text: Where did you find that equation? Can you please links us to the source of that equation?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-18T02:00:23.513Z
Comments.Text: @nbro I thought that is the TD error feed back for retrain the policy network and critic network.
Comments.UserDisplayName: lserlohn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-18T02:01:59.637Z
Comments.Text: it is from https://www.slideshare.net/zhihua98/actor-critic-algorithm in page 11
Comments.UserDisplayName: lserlohn
Tags: <reinforcement-learning><actor-critic-methods>
Title: How the actor use the output from the critic to make action in actor-critic network?
UserDisplayName: lserlohn
ThreadText: # How the actor use the output from the critic to make action in actor-critic network?
  
  (Asked by: username_0 on 2019-02-16)
  
  I am reading about the actor-critic architecture. I am confused about how the actor determines the action using the value (or future reward) from the critic network.
  
  Below you have the most popular picture of actor-critic network.
  
  [![Actor Critic Network](https://i.sstatic.net/pKKmS.png)](https://i.sstatic.net/pKKmS.png)
  
  It looks like the input of the actor network is only the "state" variable ($s\_t$), it has nothing to do with the critic network.
  
  However, from the equation below
  
  [![Policy Gradient with Critic Network](https://i.sstatic.net/OKy4l.png)](https://i.sstatic.net/OKy4l.png)
  
  the actor seems to be related to critic network.
  
  I have a few questions
  
  1. Does actor network has two inputs, state variable and future reward (output from critic network), or only the state variable?
  2. If the actor network does take future reward as input, how does it use it? Only during the training stage, or in the action making stage?
  3. Is there a "policy iteration" procedure happens during decision making stage, i.e. for every state $s\_t$, policy network will make several attempts with critic network, and output the best policy?
  
  **Tags:** reinforcement-learning, actor-critic-methods
  > **Comments:**
  > **username_1**: "It looks like the input of the actor network is only the "state" variable (), it has nothing to do with the critic network.". Note that the picture contains an arrow from the critic which crosses the the actor.  
  > **username_1**: Where did you find that equation? Can you please links us to the source of that equation?  
  > **username_0**: @nbro I thought that is the TD error feed back for retrain the policy network and critic network.  
  > **username_0**: it is from https://www.slideshare.net/zhihua98/actor-critic-algorithm in page 11  
  
  ---
  
  ## 0 answers
  
  