Answers.Body: <p>When using the loss function for the critic described in your question, the Actor-Critic is an <strong>on-policy</strong> approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to be quite close, but the notation/terminology is not quite on point.</p>
  
  <p>First it's important to realize that the <span class="math-container">$Q(s, a)$</span> critic is an <strong>estimator</strong>, we're training it to <strong>estimate</strong> state-action values. You could say that we are training it such that it can hopefully provide accurate estimates of:</p>
  
  <p><span class="math-container">$$Q_w^{\pi} (s_t, a_t) \approx \mathbb{E}_{\pi} \left[ r_t + \gamma V^{\pi}(s_{t+1}) \right].$$</span></p>
  
  <p>You'll notice I've added quite a number of symbols there in comparison to your <span class="math-container">$Q(s_t, a_t) = r_t + V(s_{t+1})$</span>:</p>
  
  <ul>
  <li>I have added the <span class="math-container">$\pi$</span> superscript to <span class="math-container">$Q$</span> and <span class="math-container">$V$</span>; this denotes the <strong>behaviour policy</strong>, which is the policy that we're using to generate experience. In on-policy methods, this is equal to the <strong>target policy</strong> (the policy for which we're learning to predict values). Adding this superscript makes explicit the fact that we're learning expected returns for states and state-action pairs that are only accurate under the assumption that we continue following the <span class="math-container">$\pi$</span> policy from state <span class="math-container">$s_{t+1}$</span> onwards.</li>
  <li>I added the discount factor <span class="math-container">$\gamma$</span>, which is probably just a tiny detail you forgot.</li>
  <li>I added <span class="math-container">$\mathbb{E}_{\pi}$</span> to indicate that we're trying to estimate an <strong>expectation under <span class="math-container">$\pi$</span></strong> (and the environment's dynamics).</li>
  </ul>
  
  <p>So, the critic is trained to estimate <span class="math-container">$Q^{\pi}(s, a)$</span>, which can intuitively be interpreted as the long-term discounted rewards that we expect to collect when executing <span class="math-container">$a$</span> in <span class="math-container">$s$</span>, and selecting actions according to the distribution <span class="math-container">$\pi$</span> subsequently. It definitely still is trying to estimate <span class="math-container">$Q(s, a)$</span> values for state-action pairs, not just <span class="math-container">$V(s)$</span> values for states alone.</p>
  
  <hr>
  
  <blockquote>
    <p><strong>What In actor critic outputs <span class="math-container">$Q(s_{t+1},a_{t+1})$</span>?</strong></p>
  </blockquote>
  
  <p>In practice, when using the loss function described in your question, <span class="math-container">$a_{t+1}$</span> really simply is a single action selected in an actual trajectory of experience by the policy <span class="math-container">$\pi$</span>. The trained network simply takes <span class="math-container">$s_{t+1}$</span> as input, and the output corresponding to <strong>a single action</strong> <span class="math-container">$a_{t+1}$</span> <strong>as selected by the policy</strong> is used as the value for <span class="math-container">$Q(s_{t+1},a_{t+1})$</span> in the update rule. </p>
  
  <p>The update rule does not involve any sum over all actions, multiplied with their probabilities. The "trick" is that we do not just run the update rule a single time, but we expect to generate lots (sometimes millions) of trajectories as experience, and we repeatedly run the update rule. In different trajectories, we'll experience the different actions <span class="math-container">$a_t$</span> as samples with approximately the correct frequencies, and <strong>in expectation</strong> we'll have proper update targets (except for potential bias resulting from function approximation).</p>
  
  <hr>
  
  <blockquote>
    <p>I want to use actor critic with experience replay in an environment with a large action space</p>
  </blockquote>
  
  <p>The Actor-Critic method in your question is, as I mentioned above, <strong>on-policy</strong>. This means that the experience used in update rules has to be generated according to exactly the same policy for which you are also learning value estimates. This is incompatible with the idea of experience replay, because old trajectories stored in a replay buffer were generated by older versions of your policy.</p>
  
  <p>There are <strong>off-policy</strong> Actor-Critic methods which can correct for the mismatch in distributions and use experience replay, but these are going to be quite a bit more complicated. Examples are <a href="https://arxiv.org/abs/1611.01224" rel="noreferrer">ACER</a> and <a href="https://arxiv.org/abs/1802.01561" rel="noreferrer">IMPALA</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T18:33:17.430Z
Answers.Text: Thanks! Just one question: to my best understanding, the critic is a function mapping from states to rewards, meaning it takes no action as input. For example, openAI's https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py and also https://medium.freecodecamp.org/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d
  That said, I may just be wrong, and the code to which I linked just calls something "critic" that is actually not a "literature critic"
  So, what IS a critic? a function from state only, or from state,action to reward?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T18:37:08.157Z
Answers.Text: The first link is to code where the critic outputs a value given a state ("value head"). The second actually shows equations for Q(s,a), but it is unclear where 'a' comes from. Also the drawings show that the Qhat doesn't take an action in, ever.
  Please expand on how to evaluate the Q(s,a) at t and t+1, and how to calculate the loss, and most importantly, what the critic maps from: states, or states,actions
  Thanks!
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T18:39:34.440Z
Answers.Text: And another confution about the same topic: https://www.researchgate.net/profile/Dewen_Hu/publication/6170058/figure/fig1/AS:349389655691266@1460312393268/Policy-iteration-and-actor-critic-learning.png shows again, the critic doesn't take an action as input. I just have o get the critic definition down
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T18:42:57.517Z
Answers.Text: @Gulzar If someone writes the critic as providing $Q(s, a)$ estimates, then it should be exactly that; something that maps from state-action pairs to values (in Deep Neural Networks often implemented as a network that takes a state vector as input, and produces multiple outputs; one per action, which can then be interpreted as the $Q(s, a)$ for the corresponding output). Actor-Critic is not one specific algorithm though. The $Q(s, a)$ can in practice be replaced by many many different things by "subtracting a state-dependent baseline".
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T18:44:55.990Z
Answers.Text: The blog post you linked to discusses one such replacement when it starts talking about the advantage function $A(s, a)$ (which is again a mapping from state-action pairs to scalars). And then that blog post also shows how the $A(s, a)$ function can again be **estimated** using only a trained $V(s)$ estimator + a reward observation $r$. Once you've arrived at that point, it's sufficient to train a neural network to provide only the estimates for $V(s)$, so then you only need a mapping from states to scalars.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T18:53:08.107Z
Answers.Text: are there instances where "critic" refers to just a function from state to reward, disregarding action?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T18:58:43.390Z
Answers.Text: I'm sorry, I'm just not getting it... Before adding in advantage, I am still stuck in Q(s,a) evaluation. OpenAi used a single output value head for 2 possible actions. Are you saying what I should do is either make a "value" output for each action, which would behave as Q(s,a) for that corresponding action, or alternatively, separate the actor and critic to two separate networks, where the critic maps state,action to reward?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T19:10:46.517Z
Answers.Text: The PyTorch example (which was not written by OpenAI) is indeed learning to output $V(s)$ estimates, it's simply a different actor-critic approach from the one described in the blog post. I'm finding the code a bit difficult to read, but if I'm not mistaken the PyTorch example code is doing what's described under ["Baselines in Policy Gradients"](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#baselines-in-policy-gradients) with $b(s_t) = V^{\pi}(s_t)$. The blog post appears to start with $\Phi_t = Q(s_t, a_t)$ under "Other Forms of the Policy Gradient" further down.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T19:12:17.997Z
Answers.Text: So you absolutely can have an actor-critic method where the critic is only trained to predict $V(s_t)$ (and then there are still multiple different possible ways in which to use it), you're just doing something different then from what is described in the beginning of the blog post. It's a different Actor-Critic method.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-06T18:07:28.307Z
Answers.LastActivityDate: 2019-02-06T18:07:28.307Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>In actor critic, <a href="https://medium.freecodecamp.org/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d" rel="nofollow noreferrer">The equations for calculating the loss in actor critic</a> are an </p>
  
  <p><em>actor</em> loss (parameterized by <span class="math-container">$\theta$</span>) </p>
  
  <p><span class="math-container">$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$</span> </p>
  
  <p>and a <em>critic</em> loss (parameterized by <span class="math-container">$w$</span>) </p>
  
  <p><span class="math-container">$$r(s_t,a_t) + \gamma Q_w(s_{t+1}, a_{t+1}) - Q_w(s_{t}, a_t).$$</span></p>
  
  <p>This is <a href="https://datascience.stackexchange.com/questions/20535/what-is-experience-replay-and-what-are-its-benefits">bootstrapping in experience replay</a>:</p>
  
  <p><span class="math-container">$$
  L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left(r + \gamma \max_{a'} Q(s', a'; \theta_i^-) - Q(s, a; \theta_i)\right)^2 \right]
  $$</span></p>
  
  <p>It is clear that bootstrapping is comparable to the critic loss, except that the <span class="math-container">$max$</span> operation is lacking from the critic.</p>
  
  <p><strong>As i see it, (correct me if I'm wrong)</strong>:</p>
  
  <p><span class="math-container">$Q(s_t,a_t) = V(s_{t+1}) + r_t$</span> where <span class="math-container">$a_t$</span> is the actual action that had been taken.</p>
  
  <p>The critic, as I understand, estimates <span class="math-container">$V(s)$</span></p>
  
  <p><strong>My question:</strong></p>
  
  <p><strong>What exactly is the critic calculating?</strong></p>
  
  <p><strong>What In actor critic outputs <span class="math-container">$Q(s_{t+1},a_{t+1})$</span>?</strong></p>
  
  <p>It seems to me like the critic calculates the average next state <span class="math-container">$s_{t+1}$</span> value, over all possible actions, with their corresponding probabilities, yielding</p>
  
  <p><span class="math-container">$Q(s_t, a_t) = r_t + \sum_{a_{t+1} \in A}P(a_{t+1}|s_t)V(s_{t+1})$</span></p>
  
  <p>Which would mean that in order to get <span class="math-container">$Q(s_{t+1}, a_{t+1})$</span> for the above formula, I would need to calculate</p>
  
  <p><span class="math-container">$Q(s_{t+1}, a_{t+1}) = r_{t+1} + \sum_{a_{t+2} \in A}P(a_{t+2}|s_{t+1})V(s_{t+2})$</span></p>
  
  <p>Where <span class="math-container">$V(s_{t+2})$</span> is the critic output on <span class="math-container">$s_{t+2}$</span>, a state we get to by taking action <span class="math-container">$a_{t+1}$</span> from state <span class="math-container">$s_{t+1}$</span> but I am not sure that is indeed the meaning of the critic output and still it is unclear to me how to get <span class="math-container">$Q(s_{t+1}, a_{t+1})$</span> from actor critic.</p>
  
  <p>If indeed that is what's being calculated, then why is it mathematically true that an improvement is being made? Or why does it make sense (even if not mathematically always true)?</p>
  
  <hr>
  
  <p>Practical use:</p>
  
  <p>I want to use actor critic with experience replay in an environment with a large action space (could be continuous). Therefore, I cannot use the <span class="math-container">$max$</span> term. I need to understand the correct equation for the critic loss, and why it works.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-06T15:48:43.263Z
Comments.Text: It is not very clear your "As i see it, (correct me if I'm wrong)" part. Can you be more precise with your notation? For example, you have not $Q(s, a)$ anywhere in the previous equations. Also, why are you trying to look at Q(s, a) as Q(s)? I've never seen Q(s), but only V(s).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-06T15:51:15.820Z
Comments.Text: Furthermore, what is the relation between your "As i see it, (correct me if I'm wrong)" part and your actual question: "What exactly is the critic calculating?"? Or, what does experience replay have to do with this? I think you should ask one question per post, otherwise it is even more confusing. Ask just one question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-06T16:46:11.023Z
Comments.Text: @nbro
  I made a major edit. Please check again to see if it is clearer now. Also, I am not confident enough about any of my understanding, so even asking the correct questions is difficult.
Comments.UserDisplayName: Gulzar
Tags: <machine-learning><reinforcement-learning><q-learning><policy-gradients><actor-critic-methods>
Title: Meaning of Actor Output in Actor Critic Reinforcement Learning
UserDisplayName: Gulzar
ThreadText: # Meaning of Actor Output in Actor Critic Reinforcement Learning
  
  (Asked by: username_0 on 2019-02-06)
  
  In actor critic, [The equations for calculating the loss in actor critic](https://medium.freecodecamp.org/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d) are an
  
  *actor* loss (parameterized by $\theta$)
  
  $$log[\pi\_\theta(s\_t,a\_t)]Q\_w(s\_t,a\_t)$$
  
  and a *critic* loss (parameterized by $w$)
  
  $$r(s\_t,a\_t) + \gamma Q\_w(s\_{t+1}, a\_{t+1}) - Q\_w(s\_{t}, a\_t).$$
  
  This is [bootstrapping in experience replay](https://datascience.stackexchange.com/questions/20535/what-is-experience-replay-and-what-are-its-benefits):
  
  $$
  L\_i(\theta\_i) = \mathbb{E}\_{(s, a, r, s') \sim U(D)} \left[ \left(r + \gamma \max\_{a'} Q(s', a'; \theta\_i^-) - Q(s, a; \theta\_i)\right)^2 \right]
  $$
  
  It is clear that bootstrapping is comparable to the critic loss, except that the $max$ operation is lacking from the critic.
  
  **As i see it, (correct me if I'm wrong)**:
  
  $Q(s\_t,a\_t) = V(s\_{t+1}) + r\_t$ where $a\_t$ is the actual action that had been taken.
  
  The critic, as I understand, estimates $V(s)$
  
  **My question:**
  
  **What exactly is the critic calculating?**
  
  **What In actor critic outputs $Q(s\_{t+1},a\_{t+1})$?**
  
  It seems to me like the critic calculates the average next state $s\_{t+1}$ value, over all possible actions, with their corresponding probabilities, yielding
  
  $Q(s\_t, a\_t) = r\_t + \sum\_{a\_{t+1} \in A}P(a\_{t+1}|s\_t)V(s\_{t+1})$
  
  Which would mean that in order to get $Q(s\_{t+1}, a\_{t+1})$ for the above formula, I would need to calculate
  
  $Q(s\_{t+1}, a\_{t+1}) = r\_{t+1} + \sum\_{a\_{t+2} \in A}P(a\_{t+2}|s\_{t+1})V(s\_{t+2})$
  
  Where $V(s\_{t+2})$ is the critic output on $s\_{t+2}$, a state we get to by taking action $a\_{t+1}$ from state $s\_{t+1}$ but I am not sure that is indeed the meaning of the critic output and still it is unclear to me how to get $Q(s\_{t+1}, a\_{t+1})$ from actor critic.
  
  If indeed that is what's being calculated, then why is it mathematically true that an improvement is being made? Or why does it make sense (even if not mathematically always true)?
  
  ---
  
  Practical use:
  
  I want to use actor critic with experience replay in an environment with a large action space (could be continuous). Therefore, I cannot use the $max$ term. I need to understand the correct equation for the critic loss, and why it works.
  
  **Tags:** machine-learning, reinforcement-learning, q-learning, policy-gradients, actor-critic-methods
  
  > **Comments:**
  > **username_1**: It is not very clear your "As i see it, (correct me if I'm wrong)" part. Can you be more precise with your notation? For example, you have not $Q(s, a)$ anywhere in the previous equations. Also, why are you trying to look at Q(s, a) as Q(s)? I've never seen Q(s), but only V(s).  
  > **username_1**: Furthermore, what is the relation between your "As i see it, (correct me if I'm wrong)" part and your actual question: "What exactly is the critic calculating?"? Or, what does experience replay have to do with this? I think you should ask one question per post, otherwise it is even more confusing. Ask just one question.  
  > **username_0**: @username_1 I made a major edit. Please check again to see if it is clearer now. Also, I am not confident enough about any of my understanding, so even asking the correct questions is difficult.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-02-06. Score: 5 | Accepted answer
  
  When using the loss function for the critic described in your question, the Actor-Critic is an **on-policy** approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to be quite close, but the notation/terminology is not quite on point.
  
  First it's important to realize that the $Q(s, a)$ critic is an **estimator**, we're training it to **estimate** state-action values. You could say that we are training it such that it can hopefully provide accurate estimates of:
  
  $$Q\_w^{\pi} (s\_t, a\_t) \approx \mathbb{E}\_{\pi} \left[ r\_t + \gamma V^{\pi}(s\_{t+1}) \right].$$
  
  You'll notice I've added quite a number of symbols there in comparison to your $Q(s\_t, a\_t) = r\_t + V(s\_{t+1})$:
  
  * I have added the $\pi$ superscript to $Q$ and $V$; this denotes the **behaviour policy**, which is the policy that we're using to generate experience. In on-policy methods, this is equal to the **target policy** (the policy for which we're learning to predict values). Adding this superscript makes explicit the fact that we're learning expected returns for states and state-action pairs that are only accurate under the assumption that we continue following the $\pi$ policy from state $s\_{t+1}$ onwards.
  * I added the discount factor $\gamma$, which is probably just a tiny detail you forgot.
  * I added $\mathbb{E}\_{\pi}$ to indicate that we're trying to estimate an **expectation under $\pi$** (and the environment's dynamics).
  
  So, the critic is trained to estimate $Q^{\pi}(s, a)$, which can intuitively be interpreted as the long-term discounted rewards that we expect to collect when executing $a$ in $s$, and selecting actions according to the distribution $\pi$ subsequently. It definitely still is trying to estimate $Q(s, a)$ values for state-action pairs, not just $V(s)$ values for states alone.
  
  ---
  
  > **What In actor critic outputs $Q(s\_{t+1},a\_{t+1})$?**
  
  In practice, when using the loss function described in your question, $a\_{t+1}$ really simply is a single action selected in an actual trajectory of experience by the policy $\pi$. The trained network simply takes $s\_{t+1}$ as input, and the output corresponding to **a single action** $a\_{t+1}$ **as selected by the policy** is used as the value for $Q(s\_{t+1},a\_{t+1})$ in the update rule.
  
  The update rule does not involve any sum over all actions, multiplied with their probabilities. The "trick" is that we do not just run the update rule a single time, but we expect to generate lots (sometimes millions) of trajectories as experience, and we repeatedly run the update rule. In different trajectories, we'll experience the different actions $a\_t$ as samples with approximately the correct frequencies, and **in expectation** we'll have proper update targets (except for potential bias resulting from function approximation).
  
  ---
  
  > I want to use actor critic with experience replay in an environment with a large action space
  
  The Actor-Critic method in your question is, as I mentioned above, **on-policy**. This means that the experience used in update rules has to be generated according to exactly the same policy for which you are also learning value estimates. This is incompatible with the idea of experience replay, because old trajectories stored in a replay buffer were generated by older versions of your policy.
  
  There are **off-policy** Actor-Critic methods which can correct for the mismatch in distributions and use experience replay, but these are going to be quite a bit more complicated. Examples are [ACER](https://arxiv.org/abs/1611.01224) and [IMPALA](https://arxiv.org/abs/1802.01561).
  
  > **Comments:**
  > **username_0**: Thanks! Just one question: to my best understanding, the critic is a function mapping from states to rewards, meaning it takes no action as input. For example, openAI's https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py and also https://medium.freecodecamp.org/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d That said, I may just be wrong, and the code to which I linked just calls something "critic" that is actually not a "literature critic" So, what IS a critic? a function from state only, or from state,action to reward?  
  > **username_0**: The first link is to code where the critic outputs a value given a state ("value head"). The second actually shows equations for Q(s,a), but it is unclear where 'a' comes from. Also the drawings show that the Qhat doesn't take an action in, ever. Please expand on how to evaluate the Q(s,a) at t and t+1, and how to calculate the loss, and most importantly, what the critic maps from: states, or states,actions Thanks!  
  > **username_0**: And another confution about the same topic: https://www.researchgate.net/profile/Dewen_Hu/publication/6170058/figure/fig1/AS:349389655691266@1460312393268/Policy-iteration-and-actor-critic-learning.png shows again, the critic doesn't take an action as input. I just have o get the critic definition down  
  > **username_2**: @username_0 If someone writes the critic as providing $Q(s, a)$ estimates, then it should be exactly that; something that maps from state-action pairs to values (in Deep Neural Networks often implemented as a network that takes a state vector as input, and produces multiple outputs; one per action, which can then be interpreted as the $Q(s, a)$ for the corresponding output). Actor-Critic is not one specific algorithm though. The $Q(s, a)$ can in practice be replaced by many many different things by "subtracting a state-dependent baseline".  
  > **username_2**: The blog post you linked to discusses one such replacement when it starts talking about the advantage function $A(s, a)$ (which is again a mapping from state-action pairs to scalars). And then that blog post also shows how the $A(s, a)$ function can again be **estimated** using only a trained $V(s)$ estimator + a reward observation $r$. Once you've arrived at that point, it's sufficient to train a neural network to provide only the estimates for $V(s)$, so then you only need a mapping from states to scalars.  
  > **username_0**: are there instances where "critic" refers to just a function from state to reward, disregarding action?  
  > **username_0**: I'm sorry, I'm just not getting it... Before adding in advantage, I am still stuck in Q(s,a) evaluation. OpenAi used a single output value head for 2 possible actions. Are you saying what I should do is either make a "value" output for each action, which would behave as Q(s,a) for that corresponding action, or alternatively, separate the actor and critic to two separate networks, where the critic maps state,action to reward?  
  > **username_2**: The PyTorch example (which was not written by OpenAI) is indeed learning to output $V(s)$ estimates, it's simply a different actor-critic approach from the one described in the blog post. I'm finding the code a bit difficult to read, but if I'm not mistaken the PyTorch example code is doing what's described under ["Baselines in Policy Gradients"](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#baselines-in-policy-gradients) with $b(s_t) = V^{\pi}(s_t)$. The blog post appears to start with $\Phi_t = Q(s_t, a_t)$ under "Other Forms of the Policy Gradient" further down.  
  > **username_2**: So you absolutely can have an actor-critic method where the critic is only trained to predict $V(s_t)$ (and then there are still multiple different possible ways in which to use it), you're just doing something different then from what is described in the beginning of the blog post. It's a different Actor-Critic method.  
  
  ---
  
  