Answers.Body: <p>Absolutely, it’s a really interesting problem. Here is a paper detailing <a href="https://arxiv.org/abs/1205.4839" rel="nofollow noreferrer">off policy actor critic</a>. This is important because this method can also support continuous actions.  </p>
  
  <p>The general idea of off-policy algorithms is to compare the actions performed by a behaviour policy (which is actually acting in the world) with the actions the target policy (the policy we want to learn) would have chosen. Using this comparison we can determine a ratio (<span class="math-container">$0 \leq \rho \leq 1$</span>) which can scale the update to the target policy by the probability of the target policy taking that action. A higher <span class="math-container">$\rho$</span>, the more alike the two policies are, and this increases the magnitude of the learning update for the target policy for that step. A <span class="math-container">$\rho$</span> of <span class="math-container">$0$</span>, and the update is ignored. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-12-24T17:35:08.393Z
Answers.LastActivityDate: 2019-11-19T01:54:37.057Z
Answers.LastEditDate: 2019-11-19T01:54:37.057Z
Answers.UserDisplayName: Jaden Travnik
Body: <p>Do off-policy policy gradient methods exist?</p>
  
  <p>I know that policy gradient methods themselves using the policy function for sampling rollouts. But can't we easily have a model for sampling from the environment? If so, I've never seen this done before.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-23T07:41:33.807Z
Comments.Text: An extension to this would be - Can we leverage experience replay concept in policy gradient algo? I understand the answer is No because we are not talking of an off policy update, but are there alternatives or workarounds
Comments.UserDisplayName: Allohvk
Tags: <reinforcement-learning><policy-gradients><off-policy-methods>
Title: Do off-policy policy gradient methods exist?
UserDisplayName: echo
ThreadText: # Do off-policy policy gradient methods exist?
  
  (Asked by: username_0 on 2017-12-23)
  
  Do off-policy policy gradient methods exist?
  
  I know that policy gradient methods themselves using the policy function for sampling rollouts. But can't we easily have a model for sampling from the environment? If so, I've never seen this done before.
  
  **Tags:** reinforcement-learning, policy-gradients, off-policy-methods
  
  > **Comments:**
  > **username_1**: An extension to this would be - Can we leverage experience replay concept in policy gradient algo? I understand the answer is No because we are not talking of an off policy update, but are there alternatives or workarounds  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-12-24. Score: 7
  
  Absolutely, it’s a really interesting problem. Here is a paper detailing [off policy actor critic](https://arxiv.org/abs/1205.4839). This is important because this method can also support continuous actions.
  
  The general idea of off-policy algorithms is to compare the actions performed by a behaviour policy (which is actually acting in the world) with the actions the target policy (the policy we want to learn) would have chosen. Using this comparison we can determine a ratio ($0 \leq \rho \leq 1$) which can scale the update to the target policy by the probability of the target policy taking that action. A higher $\rho$, the more alike the two policies are, and this increases the magnitude of the learning update for the target policy for that step. A $\rho$ of $0$, and the update is ignored.
  
  
  ---
  
  