<p>Absolutely, it’s a really interesting problem. Here is a paper detailing <a href="https://arxiv.org/abs/1205.4839" rel="nofollow noreferrer">off policy actor critic</a>. This is important because this method can also support continuous actions.  </p>

<p>The general idea of off-policy algorithms is to compare the actions performed by a behaviour policy (which is actually acting in the world) with the actions the target policy (the policy we want to learn) would have chosen. Using this comparison we can determine a ratio (<span class="math-container">$0 \leq \rho \leq 1$</span>) which can scale the update to the target policy by the probability of the target policy taking that action. A higher <span class="math-container">$\rho$</span>, the more alike the two policies are, and this increases the magnitude of the learning update for the target policy for that step. A <span class="math-container">$\rho$</span> of <span class="math-container">$0$</span>, and the update is ignored. </p>

CC BY-SA 4.0
2017-12-24T17:35:08.393Z
2019-11-19T01:54:37.057Z
2019-11-19T01:54:37.057Z
Jaden Travnik
<p>Do off-policy policy gradient methods exist?</p>

<p>I know that policy gradient methods themselves using the policy function for sampling rollouts. But can't we easily have a model for sampling from the environment? If so, I've never seen this done before.</p>

CC BY-SA 4.0
2022-11-23T07:41:33.807Z
An extension to this would be - Can we leverage experience replay concept in policy gradient algo? I understand the answer is No because we are not talking of an off policy update, but are there alternatives or workarounds
Allohvk
<reinforcement-learning><policy-gradients><off-policy-methods>
Do off-policy policy gradient methods exist?
echo
# Do off-policy policy gradient methods exist?

(Asked by: username_0 on 2017-12-23)

Do off-policy policy gradient methods exist?

I know that policy gradient methods themselves using the policy function for sampling rollouts. But can't we easily have a model for sampling from the environment? If so, I've never seen this done before.

**Tags:** reinforcement-learning, policy-gradients, off-policy-methods

> **Comments:**
> **username_1**: An extension to this would be - Can we leverage experience replay concept in policy gradient algo? I understand the answer is No because we are not talking of an off policy update, but are there alternatives or workarounds  

---

## 1 answers

### Answer by username_2 on 2017-12-24. Score: 7

Absolutely, it’s a really interesting problem. Here is a paper detailing [off policy actor critic](https://arxiv.org/abs/1205.4839). This is important because this method can also support continuous actions.

The general idea of off-policy algorithms is to compare the actions performed by a behaviour policy (which is actually acting in the world) with the actions the target policy (the policy we want to learn) would have chosen. Using this comparison we can determine a ratio ($0 \leq \rho \leq 1$) which can scale the update to the target policy by the probability of the target policy taking that action. A higher $\rho$, the more alike the two policies are, and this increases the magnitude of the learning update for the target policy for that step. A $\rho$ of $0$, and the update is ignored.


---

