Answers.Body: <p>DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be expressed as</p>
  <p><span class="math-container">$$\nabla _{\theta^\mu} J \approx \mathbb{E}_{s_t \sim \rho^\beta} \left[ \nabla _{\theta^\mu} Q(s,a|\theta^Q) | s=s_t, a=\mu(s_t ; \theta ^\mu)  \right]\;.$$</span></p>
  <p>We are interested in learning about the policy parameters of <span class="math-container">$\mu$</span>, denoted by <span class="math-container">$\theta$</span>, but we take expected with respect to some discounted state distribution induced by a policy <span class="math-container">$\beta$</span>, which we will denote as <span class="math-container">$\rho^\beta$</span>.</p>
  <p>To summarise, we are learning off-policy as the expectation of the gradient is taken with respect to some state distribution that occurs under some policy that we are not learning about.</p>
  <p>Given that on-policy learning is a special case of off-policy learning, if the replay buffer had a size of one, i.e. we use only the most recent experience tuple to perform parameter updates, then DDPG would be on-policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-20T10:59:13.013Z
Answers.LastActivityDate: 2021-05-09T23:13:58.287Z
Answers.LastEditDate: 2021-05-09T23:13:58.287Z
Answers.UserDisplayName: David
Answers.Body: <p>If there was no action noise it would probably not explore enough to obtain a good estimate of Q or the policy gradient.</p>
  
  <p>Instead of estimating Q of the target policy you could estimate Q of the behavior policy but then you have a stochastic policy and the deterministic policy gradient theorem does not work anymore as it is a special case of the stochastic policy gradient theorem (see Section 3.3 of the DPG paper, <a href="http://proceedings.mlr.press/v32/silver14.pdf" rel="nofollow noreferrer">http://proceedings.mlr.press/v32/silver14.pdf</a>). You would have to use policy gradient theorem of Section 2.2 from the DPG paper.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-30T04:22:19.173Z
Answers.Text: Thanks for your answer, what is the behaviour policy in DDPG?
Answers.UserDisplayName: GoingMyWay
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-30T07:27:12.353Z
Answers.Text: In algorithm 1 from the DDPG paper (https://arxiv.org/pdf/1509.02971.pdf) you find the behavior policy in line 8 (Select action a_t ...). It is the target policy plus some exploration noise. For this paper they used an "Ornstein-Uhlenbeck process" (last paragraph of Section 3 and Appendix 7). It seems like Gaussian noise works, too (source: https://spinningup.openai.com/en/latest/algorithms/ddpg.html#exploration-vs-exploitation).
Answers.UserDisplayName: alfa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-30T12:58:01.440Z
Answers.Text: I think that is not a behaviour policy right? Even on-policy algorithms like PPO has such policy.
Answers.UserDisplayName: GoingMyWay
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T14:21:04.630Z
Answers.Text: Yes, it is. In on-policy algorithms the behavior policy is the target policy. The behavior policy is used for exploration and the target policy is used to approximate the value function (Q, V, or A). See Section 5.5 of Sutton & Barto (http://incompleteideas.net/book/RLbook2020.pdf).
Answers.UserDisplayName: alfa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T14:25:27.697Z
Answers.Text: Maybe you have a wrong idea about what off-policy and on-policy means. What would you say is the difference between both?
Answers.UserDisplayName: alfa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-02T01:12:14.387Z
Answers.Text: I don't think adding noise to actions is a behaviour policy, if that so, in PPO (on-policy), adding a noise to its actions makes it off-policy? It is not the definition. The behaviour policy can be a bad policy, it is used to collect data for target policy to learn. I do not think the main purpose of behaviour policy is used for exploration.
Answers.UserDisplayName: GoingMyWay
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-05T00:05:16.660Z
Answers.Text: The behavior policy is called behavior policy because it defines how the agent behaves. The DDPG agent creates its actions through mu + noise while it approximates Q of mu - its target policy. PPO in its original form uses a stochastic policy and is on-policy. Adding noise would not change that. The stochastic policy ensures sufficient exploration. The purpose of a policy is either eploration or exploration (or both). A behavior policy is clearly more on the exploration side than the target policy.
Answers.UserDisplayName: alfa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-05T01:47:33.380Z
Answers.Text: Thanks for the reply of the purpose of adding the noise, back to my question: in DDPG, if there are no -greedy and no action noise, is DDPG an on-policy algorithm? I am not sure on this point. And I am not clear on if the behaviour is clearly more on the exploration side because I did not find theoretical guarantees on its exploration ability of the behaviour so far. If the behaviour policy is far away from the target policy, it can be a bad policy, some algorithms like Retrace/Vtrace still own some theoretical guarantees on the convergence of the target policy.
Answers.UserDisplayName: GoingMyWay
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-05T15:34:27.597Z
Answers.Text: OK, you could say that without exploration noise it is on-policy (with a deterministic policy). It would most likely not work though. If you had an infinite amount of samples, infinite capacity of the neural networks, and infinite training time for your networks you would perfectly fit Q of mu and I guess mu would get stuck in some local optimum. Theoretically this is not much different from tabular Q-learning.
Answers.UserDisplayName: alfa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-05T15:34:33.873Z
Answers.Text: There are no guarantees in Deep RL for anything anyway though, so maybe it works even better than some noisy behavior policies. ;) As far as I understand (I might be wrong here) V-trace guarantees convergence of the value function to "something between target and behavior policy" in the tabular case. With a neural network in the mix I am not sure if anything is guaranteed. V-trace, however, needs a stochastic policy to compute importance sampling weights. mu in DDPG is not stochastic. By the way, in the V-trace paper mu is the behavior policy and pi is the target policy. Another notation.
Answers.UserDisplayName: alfa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-20T12:36:28.047Z
Answers.Text: I think that although this does not answer the posed question directly, it is the correct answer. DDPG without noise is at best a partially functioning RL algorithm, and probably not of much interest. It may work successfully in environments with stochastic overlapping state transitions. TD Gammon for example got away with only greedy action choice in a SARSA/Q-learning algorithm due to randomness inherent in the game
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-29T16:39:57.343Z
Answers.LastActivityDate: 2020-04-29T17:43:34.243Z
Answers.LastEditDate: 2020-04-29T17:43:34.243Z
Answers.UserDisplayName: alfa
Body: <p>In DDPG, if there are no <span class="math-container">$\epsilon$</span>-greedy and no action noise, is DDPG an on-policy algorithm?</p>
  
Tags: <reinforcement-learning><deep-rl><ddpg><off-policy-methods><on-policy-methods>
Title: Why is DDPG an off-policy RL algorithm?
UserDisplayName: GoingMyWay
ThreadText: # Why is DDPG an off-policy RL algorithm?
  
  (Asked by: username_0 on 2020-04-29)
  
  In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
  
  **Tags:** reinforcement-learning, deep-rl, ddpg, off-policy-methods, on-policy-methods
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-08-20. Score: 3
  
  DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be expressed as
  
  $$\nabla \_{\theta^\mu} J \approx \mathbb{E}\_{s\_t \sim \rho^\beta} \left[ \nabla \_{\theta^\mu} Q(s,a|\theta^Q) | s=s\_t, a=\mu(s\_t ; \theta ^\mu) \right]\;.$$
  
  We are interested in learning about the policy parameters of $\mu$, denoted by $\theta$, but we take expected with respect to some discounted state distribution induced by a policy $\beta$, which we will denote as $\rho^\beta$.
  
  To summarise, we are learning off-policy as the expectation of the gradient is taken with respect to some state distribution that occurs under some policy that we are not learning about.
  
  Given that on-policy learning is a special case of off-policy learning, if the replay buffer had a size of one, i.e. we use only the most recent experience tuple to perform parameter updates, then DDPG would be on-policy.
  
  
  ---
  
  ### Answer by username_2 on 2020-04-29. Score: 0
  
  If there was no action noise it would probably not explore enough to obtain a good estimate of Q or the policy gradient.
  
  Instead of estimating Q of the target policy you could estimate Q of the behavior policy but then you have a stochastic policy and the deterministic policy gradient theorem does not work anymore as it is a special case of the stochastic policy gradient theorem (see Section 3.3 of the DPG paper, <http://proceedings.mlr.press/v32/silver14.pdf>). You would have to use policy gradient theorem of Section 2.2 from the DPG paper.
  
  > **Comments:**
  > **username_0**: Thanks for your answer, what is the behaviour policy in DDPG?  
  > **username_2**: In algorithm 1 from the DDPG paper (https://arxiv.org/pdf/1509.02971.pdf) you find the behavior policy in line 8 (Select action a_t ...). It is the target policy plus some exploration noise. For this paper they used an "Ornstein-Uhlenbeck process" (last paragraph of Section 3 and Appendix 7). It seems like Gaussian noise works, too (source: https://spinningup.openai.com/en/latest/algorithms/ddpg.html#exploration-vs-exploitation).  
  > **username_0**: I think that is not a behaviour policy right? Even on-policy algorithms like PPO has such policy.  
  > **username_2**: Yes, it is. In on-policy algorithms the behavior policy is the target policy. The behavior policy is used for exploration and the target policy is used to approximate the value function (Q, V, or A). See Section 5.5 of Sutton & Barto (http://incompleteideas.net/book/RLbook2020.pdf).  
  > **username_2**: Maybe you have a wrong idea about what off-policy and on-policy means. What would you say is the difference between both?  
  > **username_0**: I don't think adding noise to actions is a behaviour policy, if that so, in PPO (on-policy), adding a noise to its actions makes it off-policy? It is not the definition. The behaviour policy can be a bad policy, it is used to collect data for target policy to learn. I do not think the main purpose of behaviour policy is used for exploration.  
  > **username_2**: The behavior policy is called behavior policy because it defines how the agent behaves. The DDPG agent creates its actions through mu + noise while it approximates Q of mu - its target policy. PPO in its original form uses a stochastic policy and is on-policy. Adding noise would not change that. The stochastic policy ensures sufficient exploration. The purpose of a policy is either eploration or exploration (or both). A behavior policy is clearly more on the exploration side than the target policy.  
  > **username_0**: Thanks for the reply of the purpose of adding the noise, back to my question: in DDPG, if there are no -greedy and no action noise, is DDPG an on-policy algorithm? I am not sure on this point. And I am not clear on if the behaviour is clearly more on the exploration side because I did not find theoretical guarantees on its exploration ability of the behaviour so far. If the behaviour policy is far away from the target policy, it can be a bad policy, some algorithms like Retrace/Vtrace still own some theoretical guarantees on the convergence of the target policy.  
  > **username_2**: OK, you could say that without exploration noise it is on-policy (with a deterministic policy). It would most likely not work though. If you had an infinite amount of samples, infinite capacity of the neural networks, and infinite training time for your networks you would perfectly fit Q of mu and I guess mu would get stuck in some local optimum. Theoretically this is not much different from tabular Q-learning.  
  > **username_2**: There are no guarantees in Deep RL for anything anyway though, so maybe it works even better than some noisy behavior policies. ;) As far as I understand (I might be wrong here) V-trace guarantees convergence of the value function to "something between target and behavior policy" in the tabular case. With a neural network in the mix I am not sure if anything is guaranteed. V-trace, however, needs a stochastic policy to compute importance sampling weights. mu in DDPG is not stochastic. By the way, in the V-trace paper mu is the behavior policy and pi is the target policy. Another notation.  
  > **username_3**: I think that although this does not answer the posed question directly, it is the correct answer. DDPG without noise is at best a partially functioning RL algorithm, and probably not of much interest. It may work successfully in environments with stochastic overlapping state transitions. TD Gammon for example got away with only greedy action choice in a SARSA/Q-learning algorithm due to randomness inherent in the game  
  
  ---
  
  