Answers.Body: <p>TLDR: it might work, but it will probably struggle, or not work at all, depending on how optimal your &quot;other&quot; policy is</p>
  <hr />
  <p>So, strictly theoretically speaking, as long as the behavioral policy explores all the state-actions involved in the optimal policy, DDPG will indeed work (in other words, if the optimal policy involves an action that your safety module excludes, don't expect DDPG to learn it)</p>
  <p>Now, we don't live in a ideal world, and we have to face the reality of the approximations made along the way; and practically, it probably won't, or at least, it will struggle a lot</p>
  <p>Indeed, DDPG is part of the Q-learning algorithms, which is, by definition, off-policy, which means that you can learn from experience coming from another policy... however, the state distribution of the behavioral will bias the Q estimate, and thus increating instabilities for the learning</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-07T10:06:55.347Z
Answers.Text: Thanks for your answer. The objective of this "other" policy in the safety module is fundamentally different from the objective of the DDPG policy. In other words, it does not aim to maximize the value defined in DDPG. It might simply be some random control signals, with nothing to do with reinforcement learning. Under this circumstance, would it be more beneficial for DDPG to completely disregard the transition records generated by this "other" policy and instead explore and learn an optimal policy on its own within the safety boundaries provided by the safety module?
Answers.UserDisplayName: JJbow
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-07T18:35:40.060Z
Answers.Text: @JJbow the behavioral policy is not forced to be following the same reward function, so should not make any difference... and the on policy scenario (DDPG is both the learning policy and the exploring policy) is always the best one
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-06T21:15:25.350Z
Answers.LastActivityDate: 2024-06-06T21:15:25.350Z
Answers.UserDisplayName: Alberto
Body: <p>Let's hypothesize a scenario where some of the records (<em>s<sub>i</sub></em>, <em>a<sub>i</sub></em>, <em>r<sub>i</sub></em>, <em>s<sub>i+1</sub></em>) in the replay buffer are generated by another completely unrelated random policy. If the DDPG algorithm still samples random minibatches from this buffer for learning as usual, would the learning process proceed successfully?</p>
  <p>Actually, there's a pre-processing stage before training DDPG in my online learning application, where another module learns the safe action range. I wonder if the transition records obtained during this stage can be used to pre-train the DDPG agent.</p>
  
Tags: <reinforcement-learning><dqn><markov-decision-process><policy-gradients><ddpg>
Title: Would the DDPG algorithm still function effectively if some transitions stored in its replay buffer are generated by a completely unrelated policy?
UserDisplayName: JJbow
ThreadText: # Would the DDPG algorithm still function effectively if some transitions stored in its replay buffer are generated by a completely unrelated policy?
  
  (Asked by: username_0 on 2024-06-06)
  
  Let's hypothesize a scenario where some of the records (*si*, *ai*, *ri*, *si+1*) in the replay buffer are generated by another completely unrelated random policy. If the DDPG algorithm still samples random minibatches from this buffer for learning as usual, would the learning process proceed successfully?
  
  Actually, there's a pre-processing stage before training DDPG in my online learning application, where another module learns the safe action range. I wonder if the transition records obtained during this stage can be used to pre-train the DDPG agent.
  
  **Tags:** reinforcement-learning, dqn, markov-decision-process, policy-gradients, ddpg
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-06. Score: 0
  
  TLDR: it might work, but it will probably struggle, or not work at all, depending on how optimal your "other" policy is
  
  ---
  
  So, strictly theoretically speaking, as long as the behavioral policy explores all the state-actions involved in the optimal policy, DDPG will indeed work (in other words, if the optimal policy involves an action that your safety module excludes, don't expect DDPG to learn it)
  
  Now, we don't live in a ideal world, and we have to face the reality of the approximations made along the way; and practically, it probably won't, or at least, it will struggle a lot
  
  Indeed, DDPG is part of the Q-learning algorithms, which is, by definition, off-policy, which means that you can learn from experience coming from another policy... however, the state distribution of the behavioral will bias the Q estimate, and thus increating instabilities for the learning
  
  > **Comments:**
  > **username_0**: Thanks for your answer. The objective of this "other" policy in the safety module is fundamentally different from the objective of the DDPG policy. In other words, it does not aim to maximize the value defined in DDPG. It might simply be some random control signals, with nothing to do with reinforcement learning. Under this circumstance, would it be more beneficial for DDPG to completely disregard the transition records generated by this "other" policy and instead explore and learn an optimal policy on its own within the safety boundaries provided by the safety module?  
  > **username_1**: @JJbow the behavioral policy is not forced to be following the same reward function, so should not make any difference... and the on policy scenario (DDPG is both the learning policy and the exploring policy) is always the best one  
  
  ---
  
  