Answers.Body: <h2>That seems to be functional.</h2>
  <p>That is a great approach, as long as you are using an <a href="https://stats.stackexchange.com/questions/184657/what-is-the-difference-between-off-policy-and-on-policy-learning">off-policy algorithm</a> (since the samples you are using to learn are not the policy currently being performed), like Q-learning.</p>
  <p>By annealing the sample rate from the optimal buffer to the regular one, you introduce noise into the network and emphasize exploration (albeit more limited). This is helpful when you (the researcher) have no access to optimal policies, but merely &quot;good&quot; ones, and you still want the network to try and improve on those.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-04T16:02:32.457Z
Answers.Text: Have you tried implementing this before?
Answers.UserDisplayName: Rui Nian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-04T16:43:16.913Z
Answers.Text: I have, with both DQN and DDPG. Worked nicely with DQN, not that well with DDPG (although I don't think it was about the transfer learning, the problem was just too complex).
Answers.UserDisplayName: BlueMoon93
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-31T10:50:05.410Z
Answers.LastActivityDate: 2018-08-31T10:50:05.410Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: BlueMoon93
Body: <p>Due to my RL algorithm having difficulties learning some control actions, I've decided to use imitation learning/apprenticeship learning to guide my RL to perform the optimal actions.  I've read a few articles on the subject and just want to confirm how to implement it.</p>
  <p><em>Do I simply sample a state <span class="math-container">$s$</span>, then perform the optimal action <span class="math-container">$a^*$</span> in that state <span class="math-container">$s$</span>, calculate the reward for the action <span class="math-container">$r$</span>, and then observe the next state <span class="math-container">$s'$</span>, and finally put that into the experience replay?</em></p>
  <p>If this is the case, I am thinking of implementing it as follows:</p>
  <ol>
  <li>Initialize the optimal replay buffer <span class="math-container">$D_O$</span></li>
  <li>Add the optimal tuple of experience <span class="math-container">$(s, a^*, r, s')$</span> into the replay buffer <span class="math-container">$D_O$</span></li>
  <li>Initialize the normal replay buffer <span class="math-container">$D_N$</span></li>
  <li>During the simulation, initially sample <span class="math-container">$(s, a^*, r, s')$</span> only from the optimal replay buffer <span class="math-container">$D_O$</span>, while populating the normal replay buffer <span class="math-container">$D_N$</span> with the simulation results.</li>
  <li>As training/learning proceeds, anneal out the use of the optimal replay buffer, and sample only from the normal replay buffer.</li>
  </ol>
  <p>Would such an architecture work?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-05T23:56:03.823Z
Comments.Text: For completeness, you may want to edit your post to include the name of the RL algorithm you were using at the time (if you still remember).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><experience-replay><imitation-learning><apprenticeship-learning>
Title: In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?
UserDisplayName: Rui Nian
ThreadText: # In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?
  
  (Asked by: username_0 on 2018-08-27)
  
  Due to my RL algorithm having difficulties learning some control actions, I've decided to use imitation learning/apprenticeship learning to guide my RL to perform the optimal actions. I've read a few articles on the subject and just want to confirm how to implement it.
  
  *Do I simply sample a state $s$, then perform the optimal action $a^\*$ in that state $s$, calculate the reward for the action $r$, and then observe the next state $s'$, and finally put that into the experience replay?*
  
  If this is the case, I am thinking of implementing it as follows:
  
  1. Initialize the optimal replay buffer $D\_O$
  2. Add the optimal tuple of experience $(s, a^\*, r, s')$ into the replay buffer $D\_O$
  3. Initialize the normal replay buffer $D\_N$
  4. During the simulation, initially sample $(s, a^\*, r, s')$ only from the optimal replay buffer $D\_O$, while populating the normal replay buffer $D\_N$ with the simulation results.
  5. As training/learning proceeds, anneal out the use of the optimal replay buffer, and sample only from the normal replay buffer.
  
  Would such an architecture work?
  
  **Tags:** reinforcement-learning, deep-rl, experience-replay, imitation-learning, apprenticeship-learning
  
  > **Comments:**
  > **username_1**: For completeness, you may want to edit your post to include the name of the RL algorithm you were using at the time (if you still remember).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-08-31. Score: 3
  
  ## That seems to be functional.
  
  That is a great approach, as long as you are using an [off-policy algorithm](https://stats.stackexchange.com/questions/184657/what-is-the-difference-between-off-policy-and-on-policy-learning) (since the samples you are using to learn are not the policy currently being performed), like Q-learning.
  
  By annealing the sample rate from the optimal buffer to the regular one, you introduce noise into the network and emphasize exploration (albeit more limited). This is helpful when you (the researcher) have no access to optimal policies, but merely "good" ones, and you still want the network to try and improve on those.
  
  > **Comments:**
  > **username_0**: Have you tried implementing this before?  
  > **username_2**: I have, with both DQN and DDPG. Worked nicely with DQN, not that well with DDPG (although I don't think it was about the transfer learning, the problem was just too complex).  
  
  ---
  
  