Answers.Body: <p>There is no 'regular' formula for calculating policy loss, the regular thing when calculating policy gradient is to multiply gradient with advantage function which can be many things. Look at section 2 of <a href="https://arxiv.org/pdf/1506.02438.pdf" rel="nofollow noreferrer">this</a> paper for coverage on basic advantage functions. Also, the expected discounted reward is the same thing as the state-action value function (Q value).</p>
  <p><span class="math-container">$$Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1:\infty}, a_{t+1:\infty}}[\sum_{l=0}^\infty \gamma^lr_{t+l}]$$</span></p>
  <p>So, the variations you posted roughly calculate the same thing.</p>
  <p>Regarding the negative sign, in policy gradient methods, we want to maximize our performance function which has the following form:</p>
  <p><span class="math-container">$$J(\theta) = \sum_\limits{\substack{a}} \pi(a \mid s, \theta)A(s, a)$$</span></p>
  <p>So, the higher our performance the better it is. When people write code, they use minimizers that minimize the objective function, but, in this case, we want to maximize it, so maximizing the objective function is the same thing as minimizing the negative objective function therefore the negative sign.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T18:15:16.350Z
Answers.Text: Thanks! However, there isn't actually a negative sign in the equations. There is an absolute value (l1_loss), which makes the sign, either positive or negative, lose its meaning. Why does it make sense to only care about the distance of the actual reward to the expected reward, but not about the direction?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T19:00:31.667Z
Answers.Text: The loss that you are refering to, judging by the code, is the value loss, that is critic loss, not actor (policy) loss. Taking the example you provided in the question, objective function of the critic is to minimize TD error which is the difference between $Q_w(s_{t},a_{t})$ and $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$ . We don't care about which of those values is bigger we only care to minimize difference between them, therefore l1 loss. You could have used l2 loss as well .
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T19:05:50.143Z
Answers.Text: I still don't see how this satisfies the equations (in the original question), in which clearly, there is significance to the sign. Is the absolute value missing from the equation?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T19:36:54.033Z
Answers.Text: Are you refering to the equation from the article where they update critic with this: $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t})$ ? If yes, then like I said we want to minimize that difference so the sign doesn't really matter, maybe sometimes we take a step in negative direction and sometimes in positive direction, but as the difference reduces we wouldn't be taking any steps, we would converge to parameter vector $w$ , it's same as basic tabular q-learning
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T20:43:25.807Z
Answers.Text: Yes, I am referring to that equation. Please explain again why the sign doesn't matter. If we don't take sign into account (and the difference is not VERY small), then it could very easily cause divergence, as we could very likely change the weights in the opposite direction
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T21:36:56.987Z
Answers.Text: divergence will happen only if you set very large step parameter $\beta$ , if step is small (less than 0.1) then divergence is unlikely to happen. If in one iteration TD target is negative and we move in negative direction and if we somehow 'overshoot' the true value, in next update iteration TD target will be positive and we will move in opposite direction.  We will keep moving until we converge to the true value. It's pretty much like linear regression and LMS algorithm, that's what we are basically doing here.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-01T07:45:00.743Z
Answers.LastActivityDate: 2020-11-01T16:27:38.797Z
Answers.LastEditDate: 2020-11-01T16:27:38.797Z
Answers.UserDisplayName: Brale
Body: <p><a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py" rel="nofollow noreferrer">This PyTorch implementation of the actor-critic algorithm</a> calculates the losses like so:</p>
  <pre><code>actor_loss = -log_prob * discounted_reward
  policy_loss = F.smooth_l1_loss(value, torch.tensor([discounted_reward]))
  </code></pre>
  <p>Both are different from <a href="https://medium.freecodecamp.org/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d" rel="nofollow noreferrer">the regular formulas</a> which are, in the case of the actor loss (parameterized by <span class="math-container">$\theta$</span>):</p>
  <p><span class="math-container">$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$</span></p>
  <p>and, in the case of the critic loss (parameterized by <span class="math-container">$w$</span>):</p>
  <p><span class="math-container">$$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t}),$$</span></p>
  <p>where <span class="math-container">$r(s_t,a_t)$</span> is the immediate reward following taking the action.</p>
  <p>For the actor, &quot;the immediate critic evaluation of the transition&quot; was replaced with &quot;the discounted reward&quot;. For the critic, the discounted evaluation of the value from the next state <span class="math-container">$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$</span> was replaced by &quot;the discounted reward&quot;. The <span class="math-container">$L_1$</span> loss is then calculated, effectively discarding the sign of the (equation) loss.</p>
  <p>Questions:</p>
  <ol>
  <li><p>Why did they make these changes?</p>
  </li>
  <li><p>Why is the sign discarded for the critic loss?</p>
  </li>
  </ol>
  
Tags: <reinforcement-learning><objective-functions><pytorch><actor-critic-methods><implementation>
Title: Why is this PyTorch implementation of the actor-critic algorithm inconsistent with the mathematical formulas?
UserDisplayName: Gulzar
ThreadText: # Why is this PyTorch implementation of the actor-critic algorithm inconsistent with the mathematical formulas?
  
  (Asked by: username_0 on 2019-01-31)
  
  [This PyTorch implementation of the actor-critic algorithm](https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py) calculates the losses like so:
  
  ```
  actor_loss = -log_prob * discounted_reward
  policy_loss = F.smooth_l1_loss(value, torch.tensor([discounted_reward]))
  
  ```
  
  Both are different from [the regular formulas](https://medium.freecodecamp.org/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d) which are, in the case of the actor loss (parameterized by $\theta$):
  
  $$log[\pi\_\theta(s\_t,a\_t)]Q\_w(s\_t,a\_t)$$
  
  and, in the case of the critic loss (parameterized by $w$):
  
  $$r(s\_t,a\_t) + \gamma Q\_w(s\_{t+1},a\_{t+1}) - Q\_w(s\_{t},a\_{t}),$$
  
  where $r(s\_t,a\_t)$ is the immediate reward following taking the action.
  
  For the actor, "the immediate critic evaluation of the transition" was replaced with "the discounted reward". For the critic, the discounted evaluation of the value from the next state $r(s\_t,a\_t) + \gamma Q\_w(s\_{t+1},a\_{t+1})$ was replaced by "the discounted reward". The $L\_1$ loss is then calculated, effectively discarding the sign of the (equation) loss.
  
  Questions:
  
  1. Why did they make these changes?
  2. Why is the sign discarded for the critic loss?
  
  **Tags:** reinforcement-learning, objective-functions, pytorch, actor-critic-methods, implementation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-01. Score: 1
  
  There is no 'regular' formula for calculating policy loss, the regular thing when calculating policy gradient is to multiply gradient with advantage function which can be many things. Look at section 2 of [this](https://arxiv.org/pdf/1506.02438.pdf) paper for coverage on basic advantage functions. Also, the expected discounted reward is the same thing as the state-action value function (Q value).
  
  $$Q^\pi(s\_t, a\_t) = \mathbb{E}\_{s\_{t+1:\infty}, a\_{t+1:\infty}}[\sum\_{l=0}^\infty \gamma^lr\_{t+l}]$$
  
  So, the variations you posted roughly calculate the same thing.
  
  Regarding the negative sign, in policy gradient methods, we want to maximize our performance function which has the following form:
  
  $$J(\theta) = \sum\_\limits{\substack{a}} \pi(a \mid s, \theta)A(s, a)$$
  
  So, the higher our performance the better it is. When people write code, they use minimizers that minimize the objective function, but, in this case, we want to maximize it, so maximizing the objective function is the same thing as minimizing the negative objective function therefore the negative sign.
  
  > **Comments:**
  > **username_0**: Thanks! However, there isn't actually a negative sign in the equations. There is an absolute value (l1_loss), which makes the sign, either positive or negative, lose its meaning. Why does it make sense to only care about the distance of the actual reward to the expected reward, but not about the direction?  
  > **username_1**: The loss that you are refering to, judging by the code, is the value loss, that is critic loss, not actor (policy) loss. Taking the example you provided in the question, objective function of the critic is to minimize TD error which is the difference between $Q_w(s_{t},a_{t})$ and $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$ . We don't care about which of those values is bigger we only care to minimize difference between them, therefore l1 loss. You could have used l2 loss as well .  
  > **username_0**: I still don't see how this satisfies the equations (in the original question), in which clearly, there is significance to the sign. Is the absolute value missing from the equation?  
  > **username_1**: Are you refering to the equation from the article where they update critic with this: $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t})$ ? If yes, then like I said we want to minimize that difference so the sign doesn't really matter, maybe sometimes we take a step in negative direction and sometimes in positive direction, but as the difference reduces we wouldn't be taking any steps, we would converge to parameter vector $w$ , it's same as basic tabular q-learning  
  > **username_0**: Yes, I am referring to that equation. Please explain again why the sign doesn't matter. If we don't take sign into account (and the difference is not VERY small), then it could very easily cause divergence, as we could very likely change the weights in the opposite direction  
  > **username_1**: divergence will happen only if you set very large step parameter $\beta$ , if step is small (less than 0.1) then divergence is unlikely to happen. If in one iteration TD target is negative and we move in negative direction and if we somehow 'overshoot' the true value, in next update iteration TD target will be positive and we will move in opposite direction.  We will keep moving until we converge to the true value. It's pretty much like linear regression and LMS algorithm, that's what we are basically doing here.  
  
  ---
  
  