Answers.Body: <p>On policy algorithms contain policy and/or value update calculations that <em>assume</em> data was generated by the current policy. Breaking that assumption will cause them to miscalculate, or not function at all without some kind of intervention.</p>
  <p>As an example SARSA has the TD target (new estimate for <span class="math-container">$q(S_t,A_t)$</span>) of:</p>
  <p><span class="math-container">$$R_{t+1} + \gamma q(S_{t+1},A_{t+1})$$</span></p>
  <p>The main problem here is the value of <span class="math-container">$A_{t+1}$</span> (there's a couple of minor issues with <span class="math-container">$S_t, A_t$</span> as well, but those also apply to off policy and are why you usually want your behaviour policy to be close to the target policy).</p>
  <p>In an experience replay table you will have historic data from when the policy was different. So the value of <span class="math-container">$A_{t+1}$</span> that might be stored may not be the one that the current policy would take. Learning from that action would skew the value table to learning about whatever policy has just been sampled, instead of more accurately improve knowledge of the current policy.</p>
  <p>To fix this, you could resample the action choice, or even use an expected q value across all actions that SARSA could currently take. That fixes things, but actually it also changes your method to off-policy Expected SARSA . . .</p>
  <p>Similar issues apply to using experience replay tables with other on policy methods. They become inaccurate if used naively, and if you fix the issues with that, you reinvent off-policy methods (when possible, not all on policy methods have an off policy equivalent)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-07T15:15:32.870Z
Answers.LastActivityDate: 2023-07-07T15:15:32.870Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>In principle replay mechanisms as HER, PER, etc cannot be applied to on-policy algorithms, like SARSA and Policy gradient as stated by @Neil Slater.</p>
  <p>Anyway, If you are willing to &quot;adapt&quot; your on-policy algorithm you can introduce <strong>importance sampling weights</strong>, <span class="math-container">$\rho$</span>, that basically weight the actions <span class="math-container">$A_t$</span> coming from another policy (say <span class="math-container">$\pi_b$</span>) compared to the current policy, <span class="math-container">$\pi$</span>. For example the gradient of the RL objective would be weighted by:
  <span class="math-container">$$\rho = \frac{\pi(a\mid s)}{\pi_b(a\mid s)}$$</span></p>
  <p>The <a href="https://arxiv.org/pdf/1611.01224.pdf" rel="nofollow noreferrer">ACER</a> algorithm basically does this (see section 3) in order to work with a replay buffer, potentially improving sample-efficiency too. Moreover, HER is suitable for <em>goal-based RL</em>, meaning that the experience tuple also includes a goal <span class="math-container">$g\in G$</span>, and so the policy should also account for <span class="math-container">$g$</span> and so the ratio <span class="math-container">$\rho$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-10T05:57:50.663Z
Answers.Text: Worth noting that when using $A_t$ to $A_{t+n}$ to calculate a value, then you don't weight $A_t$ and if you are using a bootstrap from your target policy then you don't weight $A_{t+n}$ either. This is why you don't see importance sampling corrections in single step Q learning.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-09T11:11:12.210Z
Answers.LastActivityDate: 2023-07-09T11:11:12.210Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I'm wondering because I don't appreciate what is wrong with just applying <a href="https://arxiv.org/abs/1707.01495" rel="nofollow noreferrer">HER</a> to an otherwise on-policy algorithm? Like if we do that will the training stability just fall apart? And if so why? My understanding is that on-policy is just a category created by humans meaning that &quot;the default algorithm doesn't do off-policy optimization&quot;. But why does that prohibit adding off-policy elements?</p>
  
Tags: <reinforcement-learning><off-policy-methods><on-policy-methods><hindsight-experience-replay>
Title: Why does HER not work with on-policy RL algorithms?
UserDisplayName: profPlum
ThreadText: # Why does HER not work with on-policy RL algorithms?
  
  (Asked by: username_0 on 2023-07-06)
  
  I'm wondering because I don't appreciate what is wrong with just applying [HER](https://arxiv.org/abs/1707.01495) to an otherwise on-policy algorithm? Like if we do that will the training stability just fall apart? And if so why? My understanding is that on-policy is just a category created by humans meaning that "the default algorithm doesn't do off-policy optimization". But why does that prohibit adding off-policy elements?
  
  **Tags:** reinforcement-learning, off-policy-methods, on-policy-methods, hindsight-experience-replay
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-07-07. Score: 2 | Accepted answer
  
  On policy algorithms contain policy and/or value update calculations that *assume* data was generated by the current policy. Breaking that assumption will cause them to miscalculate, or not function at all without some kind of intervention.
  
  As an example SARSA has the TD target (new estimate for $q(S\_t,A\_t)$) of:
  
  $$R\_{t+1} + \gamma q(S\_{t+1},A\_{t+1})$$
  
  The main problem here is the value of $A\_{t+1}$ (there's a couple of minor issues with $S\_t, A\_t$ as well, but those also apply to off policy and are why you usually want your behaviour policy to be close to the target policy).
  
  In an experience replay table you will have historic data from when the policy was different. So the value of $A\_{t+1}$ that might be stored may not be the one that the current policy would take. Learning from that action would skew the value table to learning about whatever policy has just been sampled, instead of more accurately improve knowledge of the current policy.
  
  To fix this, you could resample the action choice, or even use an expected q value across all actions that SARSA could currently take. That fixes things, but actually it also changes your method to off-policy Expected SARSA . . .
  
  Similar issues apply to using experience replay tables with other on policy methods. They become inaccurate if used naively, and if you fix the issues with that, you reinvent off-policy methods (when possible, not all on policy methods have an off policy equivalent)
  
  
  ---
  
  ### Answer by username_2 on 2023-07-09. Score: 2
  
  In principle replay mechanisms as HER, PER, etc cannot be applied to on-policy algorithms, like SARSA and Policy gradient as stated by @Neil Slater.
  
  Anyway, If you are willing to "adapt" your on-policy algorithm you can introduce **importance sampling weights**, $\rho$, that basically weight the actions $A\_t$ coming from another policy (say $\pi\_b$) compared to the current policy, $\pi$. For example the gradient of the RL objective would be weighted by:
  $$\rho = \frac{\pi(a\mid s)}{\pi\_b(a\mid s)}$$
  
  The [ACER](https://arxiv.org/pdf/1611.01224.pdf) algorithm basically does this (see section 3) in order to work with a replay buffer, potentially improving sample-efficiency too. Moreover, HER is suitable for *goal-based RL*, meaning that the experience tuple also includes a goal $g\in G$, and so the policy should also account for $g$ and so the ratio $\rho$.
  
  > **Comments:**
  > **username_1**: Worth noting that when using $A_t$ to $A_{t+n}$ to calculate a value, then you don't weight $A_t$ and if you are using a bootstrap from your target policy then you don't weight $A_{t+n}$ either. This is why you don't see importance sampling corrections in single step Q learning.  
  
  ---
  
  