Answers.Body: <blockquote>
  <p>What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?</p>
  </blockquote>
  <p>You certainly can, that is indeed one of the advantages of off-policy learning algorithms; they're still &quot;correct&quot;, regardless of which policy generated the data that you're learning from (and a human expert providing the experience to learn from can also be viewed as such a &quot;policy&quot;).</p>
  <p>There are potential issues to be aware of though. For example, if you just put some expert-generated data in there and don't allow your agent to explore by itself, the experiences that you can learn from may be quite limited in the parts of the state-action space that they explore. So if your expert does not sufficiently explore the entire space, you cannot expect the agent to learn how to act if for whatever reason it ever ends up in some unexplored space. This is no different from what would happen if you trained with an agent that had too little exploration (like a greedy agent).</p>
  <blockquote>
  <p>Or I should go with the <em>behavior cloning</em> technique to train the actor network only, so that it converges rapidly?</p>
  </blockquote>
  <p>I cannot confidently say which approach would work better, so I cannot really answer this... I imagine the answer may also be different for specific different problem domains. But the basic principle of learning from expert data with an off-policy algorithm is not inherently wrong.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-01T13:16:50.177Z
Answers.Text: Thank you for the answer. You are right when you say that I should disable exploration when I am putting expert data in replay buffer. When I first ran the agent with this setup, I forgot to disable the exploration part and the agent slowly drifted to bad policy from a policy which was similar to expert.
Answers.UserDisplayName: Dheerendra Singh Tomar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-31T12:00:11.970Z
Answers.LastActivityDate: 2021-01-31T12:00:11.970Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after some time). So, it is getting difficult for me to continue training in this environment.</p>
  <p>The continuous state-space is <span class="math-container">$\mathcal{S}  \subseteq \mathbb{R}^{10}$</span> and the continuous action-space <span class="math-container">$\mathcal{A} \subseteq \mathbb{R}^{2}$</span>.</p>
  <p>What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?</p>
  <p>Or I should go with the <em>behavior cloning</em> technique to train the actor-network only, so that it converges rapidly?</p>
  <p>I just want to get the work done first and then I can think of exploring the environment.</p>
  
Tags: <reinforcement-learning><ddpg><off-policy-methods><experience-replay><behavioral-cloning>
Title: Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?
UserDisplayName: Dheerendra Singh Tomar
ThreadText: # Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?
  
  (Asked by: username_0 on 2021-01-27)
  
  I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after some time). So, it is getting difficult for me to continue training in this environment.
  
  The continuous state-space is $\mathcal{S} \subseteq \mathbb{R}^{10}$ and the continuous action-space $\mathcal{A} \subseteq \mathbb{R}^{2}$.
  
  What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?
  
  Or I should go with the *behavior cloning* technique to train the actor-network only, so that it converges rapidly?
  
  I just want to get the work done first and then I can think of exploring the environment.
  
  **Tags:** reinforcement-learning, ddpg, off-policy-methods, experience-replay, behavioral-cloning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-31. Score: 3 | Accepted answer
  
  > What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?
  
  You certainly can, that is indeed one of the advantages of off-policy learning algorithms; they're still "correct", regardless of which policy generated the data that you're learning from (and a human expert providing the experience to learn from can also be viewed as such a "policy").
  
  There are potential issues to be aware of though. For example, if you just put some expert-generated data in there and don't allow your agent to explore by itself, the experiences that you can learn from may be quite limited in the parts of the state-action space that they explore. So if your expert does not sufficiently explore the entire space, you cannot expect the agent to learn how to act if for whatever reason it ever ends up in some unexplored space. This is no different from what would happen if you trained with an agent that had too little exploration (like a greedy agent).
  
  > Or I should go with the *behavior cloning* technique to train the actor network only, so that it converges rapidly?
  
  I cannot confidently say which approach would work better, so I cannot really answer this... I imagine the answer may also be different for specific different problem domains. But the basic principle of learning from expert data with an off-policy algorithm is not inherently wrong.
  
  > **Comments:**
  > **username_0**: Thank you for the answer. You are right when you say that I should disable exploration when I am putting expert data in replay buffer. When I first ran the agent with this setup, I forgot to disable the exploration part and the agent slowly drifted to bad policy from a policy which was similar to expert.  
  
  ---
  
  