Answers.Body: <p>I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:</p>
  
  <p>In the <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">PPO paper</a>, after describing the standard policy gradient objective <span class="math-container">$L^{PG}$</span>, they mention the following:</p>
  
  <blockquote>
    <p>While it is appealing to perform multiple steps of optimization on this loss <span class="math-container">$L^{PG}$</span> using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates</p>
  </blockquote>
  
  <p>This is because, as soon as you've performed one update using a trajectory generated with the previous policy, you land in an <strong>off-policy</strong> situation; the experience gained in that trajectory is no longer representative of your current policy, and all the estimators (like the advantage estimator) technically become incorrect.</p>
  
  <p>With importance sampling, you can correct for this. This is also commonly used in multi-step off-policy value learning algorithms. Intuitively, the importance sampling term emphasizes estimates of advantage <span class="math-container">$\hat{A}_t$</span> corresponding to actions <span class="math-container">$a_t$</span> that have become <strong>more likely</strong> in the new policy relative to the old policy, and it de-emphasizes advantages corresponding to actions that have already become <strong>less likely</strong> in the new policy relative to the old policy. </p>
  
  <p>If an action <span class="math-container">$a_t$</span> in the old trajectory has already become highly unlikely since that trajectory of experience was generated, we have <span class="math-container">$\pi_{\theta} (a_t \vert s_t) &lt; \pi_{\theta_{\text{old}}} (a_t \vert s_t)$</span>, which means that <span class="math-container">$\frac{\pi_{\theta} (a_t \vert s_t)}{\pi_{\theta_{\text{old}}} (a_t \vert s_t)}$</span> becomes close to <span class="math-container">$0$</span>, which means that we'll reduce the influence of that particular chunk of experience on our subsequent updates. This makes sense because, due to previous updates since the generation of that trajectory, that particular part of the trajectory has already become highly unlikely anyway, and should therefore no longer be relevant for our updates.</p>
  
  <p>The ability to perform multiple updates using the same (old) trajectory anyway is useful because this increases sample-efficiency, we can re-use the same samples of experience more than once rather than using them once and then discarding them again.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-24T10:08:59.033Z
Answers.LastActivityDate: 2019-05-02T18:48:16.067Z
Answers.LastEditDate: 2019-05-02T18:48:16.067Z
Answers.UserDisplayName: Dennis Soemers
Answers.Body: <p>For everybody getting here from google, like me: the <span class="math-container">$\log$</span> might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am wrong):</p>
  <p><span class="math-container">$$\begin{aligned}
  \nabla_{\theta} L^{P G}(\theta) &amp;=\nabla_{\theta} \hat{E}_{t}\left[\log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\right] \\
  &amp;=\hat{E}_{t}\left[\nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\right]
  \end{aligned}$$</span></p>
  <p>and</p>
  <p><span class="math-container">$$
  \begin{aligned}
  \nabla_{\theta} L^{I S}(\theta)=&amp; \nabla_{\theta} \hat{E}_{t} \left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \hat{A}_{t}\right] \\
  &amp;=\hat{E}_{t} \left[\nabla_{\theta} \frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \hat{A}_{t}\right] \\
  &amp;=\hat{E}_{t} \left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \frac{\nabla_{\theta} \pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta}\left(a_{t} \mid s_{t}\right)} \hat{A}_{t}\right] \\
  &amp;=\hat{E}_{t}\left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\right]
  \end{aligned}
  $$</span></p>
  <p>So, the <span class="math-container">$\pi_{\theta}\left(a_{t} \mid s_{t}\right) $</span> in the PG function was replaced with <span class="math-container">$\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} $</span> whose derivate is the same as the log of the PG function (apart from the proportionality factor).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-11T20:44:25.983Z
Answers.Text: Hi, nice intuition, but I think there is an error with your derivation on the 4th step in the second equation. When you take the $\nabla_{\theta} \pi_{\theta}(a_t|s_t)$, why isn't the derivative taken for all the other $\pi_{\theta}(a_t|s_t)$?
Answers.UserDisplayName: Costa Huang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-12T00:26:11.057Z
Answers.Text: Nvm, I think your solution checks out.
Answers.UserDisplayName: Costa Huang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-07T01:25:42.330Z
Answers.Text: Yep, so basically the $log$ is gone because importance sampling cancels it out with the gradient of $log$.
Answers.UserDisplayName: BreakDS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-05T13:16:05.383Z
Answers.LastActivityDate: 2020-07-18T15:08:05.990Z
Answers.LastEditDate: 2020-07-18T15:08:05.990Z
Answers.UserDisplayName: mglss
Body: <p>In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients</p>
  <p><span class="math-container">$$L^{PG}(\theta) = \hat{\mathbb{E}}_t[\log \pi_{\theta}(a_t | s_t)\hat{A}_t],$$</span></p>
  <p>with the importance sampling term of the policy output probability over the old policy output probability</p>
  <p><span class="math-container">$$L^{IS}_{\theta_{old}}(\theta) = \hat{\mathbb{E}}_t \left[\frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}\hat{A}_t \right]$$</span></p>
  <p>Could someone please explain this step to me?</p>
  <p>I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the <span class="math-container">$\pi_{\theta_{old}}$</span> increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate). I'm just not sure of the reasons behind including this term in the first place.</p>
  
Tags: <reinforcement-learning><deep-rl><proximal-policy-optimization><importance-sampling><trust-region-policy-optimization>
Title: Why is the log probability replaced with the importance sampling in the loss function?
UserDisplayName: Mark
ThreadText: # Why is the log probability replaced with the importance sampling in the loss function?
  
  (Asked by: username_0 on 2018-08-23)
  
  In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients
  
  $$L^{PG}(\theta) = \hat{\mathbb{E}}\_t[\log \pi\_{\theta}(a\_t | s\_t)\hat{A}\_t],$$
  
  with the importance sampling term of the policy output probability over the old policy output probability
  
  $$L^{IS}\_{\theta\_{old}}(\theta) = \hat{\mathbb{E}}\_t \left[\frac{\pi\_{\theta}(a\_t | s\_t)}{\pi\_{\theta\_{old}}(a\_t | s\_t)}\hat{A}\_t \right]$$
  
  Could someone please explain this step to me?
  
  I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the $\pi\_{\theta\_{old}}$ increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate). I'm just not sure of the reasons behind including this term in the first place.
  
  **Tags:** reinforcement-learning, deep-rl, proximal-policy-optimization, importance-sampling, trust-region-policy-optimization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-08-24. Score: 6
  
  I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:
  
  In the [PPO paper](https://arxiv.org/abs/1707.06347), after describing the standard policy gradient objective $L^{PG}$, they mention the following:
  
  > While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates
  
  This is because, as soon as you've performed one update using a trajectory generated with the previous policy, you land in an **off-policy** situation; the experience gained in that trajectory is no longer representative of your current policy, and all the estimators (like the advantage estimator) technically become incorrect.
  
  With importance sampling, you can correct for this. This is also commonly used in multi-step off-policy value learning algorithms. Intuitively, the importance sampling term emphasizes estimates of advantage $\hat{A}\_t$ corresponding to actions $a\_t$ that have become **more likely** in the new policy relative to the old policy, and it de-emphasizes advantages corresponding to actions that have already become **less likely** in the new policy relative to the old policy.
  
  If an action $a\_t$ in the old trajectory has already become highly unlikely since that trajectory of experience was generated, we have $\pi\_{\theta} (a\_t \vert s\_t) < \pi\_{\theta\_{\text{old}}} (a\_t \vert s\_t)$, which means that $\frac{\pi\_{\theta} (a\_t \vert s\_t)}{\pi\_{\theta\_{\text{old}}} (a\_t \vert s\_t)}$ becomes close to $0$, which means that we'll reduce the influence of that particular chunk of experience on our subsequent updates. This makes sense because, due to previous updates since the generation of that trajectory, that particular part of the trajectory has already become highly unlikely anyway, and should therefore no longer be relevant for our updates.
  
  The ability to perform multiple updates using the same (old) trajectory anyway is useful because this increases sample-efficiency, we can re-use the same samples of experience more than once rather than using them once and then discarding them again.
  
  
  ---
  
  ### Answer by username_2 on 2019-07-05. Score: 4
  
  For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am wrong):
  
  $$\begin{aligned}
  \nabla\_{\theta} L^{P G}(\theta) &=\nabla\_{\theta} \hat{E}\_{t}\left[\log \pi\_{\theta}\left(a\_{t} \mid s\_{t}\right) \hat{A}\_{t}\right] \\
  &=\hat{E}\_{t}\left[\nabla\_{\theta} \log \pi\_{\theta}\left(a\_{t} \mid s\_{t}\right) \hat{A}\_{t}\right]
  \end{aligned}$$
  
  and
  
  $$
  \begin{aligned}
  \nabla\_{\theta} L^{I S}(\theta)=& \nabla\_{\theta} \hat{E}\_{t} \left[\frac{\pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)}{\pi\_{\theta\_{\text {old}}}\left(a\_{t} \mid s\_{t}\right)} \hat{A}\_{t}\right] \\
  &=\hat{E}\_{t} \left[\nabla\_{\theta} \frac{\pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)}{\pi\_{\theta\_{\text {old}}}\left(a\_{t} \mid s\_{t}\right)} \hat{A}\_{t}\right] \\
  &=\hat{E}\_{t} \left[\frac{\pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)}{\pi\_{\theta\_{\text {old}}}\left(a\_{t} \mid s\_{t}\right)} \frac{\nabla\_{\theta} \pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)}{\pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)} \hat{A}\_{t}\right] \\
  &=\hat{E}\_{t}\left[\frac{\pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)}{\pi\_{\theta\_{\text {old}}}\left(a\_{t} \mid s\_{t}\right)} \nabla\_{\theta} \log \pi\_{\theta}\left(a\_{t} \mid s\_{t}\right) \hat{A}\_{t}\right]
  \end{aligned}
  $$
  
  So, the $\pi\_{\theta}\left(a\_{t} \mid s\_{t}\right) $ in the PG function was replaced with $\frac{\pi\_{\theta}\left(a\_{t} \mid s\_{t}\right)}{\pi\_{\theta\_{\text {old}}}\left(a\_{t} \mid s\_{t}\right)} $ whose derivate is the same as the log of the PG function (apart from the proportionality factor).
  
  > **Comments:**
  > **username_3**: Hi, nice intuition, but I think there is an error with your derivation on the 4th step in the second equation. When you take the $\nabla_{\theta} \pi_{\theta}(a_t|s_t)$, why isn't the derivative taken for all the other $\pi_{\theta}(a_t|s_t)$?  
  > **username_3**: Nvm, I think your solution checks out.  
  > **username_4**: Yep, so basically the $log$ is gone because importance sampling cancels it out with the gradient of $log$.  
  
  ---
  
  