Answers.Body: <h2>Using suboptimal demonstrations in Imitation Learning</h2>
  <p>The paper you link above uses Behavior Cloning (supervised learning on the expert state-action pairs), which is known to suffer from the &quot;DAgger problem&quot; like many other imitation learning algorithms.</p>
  <p>In essence, a policy trained under imitation learning from expert demonstrations will inevitably make some suboptimal actions during the episode, and these suboptimal actions lead to states that are increasingly different from states in the expert demonstrations, which leads to more suboptimal actions.</p>
  <p>So there are cases where adding noise or perturbation to your demonstration set can lead to a more robust imitation policy[1].</p>
  <p>[1] T-Rex <a href="https://arxiv.org/abs/1904.06387" rel="nofollow noreferrer">https://arxiv.org/abs/1904.06387</a></p>
  <h2>Imitation Learning in PRIMAL</h2>
  <p>As I understand it, the paper you link to only uses expert demonstrations during imitation learning. The suboptimal demonstrations are RL samples being used for an RL objective.</p>
  <p>The authors describe their choice to combine IL and RL as follows:</p>
  <blockquote>
  <p>combining RL and IL has been shown to lead to faster, more stable
  training as well as higher-quality solutions in robot manipulation.
  These advantages are likely due to the fact that IL can help to
  quickly identify high-quality regions of the agents state-action
  space, while RL can further improve the policy by freely exploring
  these regions</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T20:45:02.730Z
Answers.LastActivityDate: 2022-11-29T20:45:02.730Z
Answers.UserDisplayName: twink_ml
Body: <p>Some IL approaches train the agents by using some specific ratio of expert demonstrations to trajectories generated using the policy being optimized.</p>
  <p>In the specific <a href="https://arxiv.org/abs/1809.03531" rel="nofollow noreferrer">paper</a> I'm reading they say <em>&quot;we experimented with various IL proportions (10-50% by increments of 10%) and observed that the RL/IL ratio does not seem to affect the performance of the trained policy by much.&quot;</em></p>
  <p>My question is: why not rely <em>only</em> on expert demonstrations instead of introducing the noise of trajectories generated by a sub-optimal policy?.</p>
  <p>My assumptions are:</p>
  <ol>
  <li>This noise helps explore the state space beyond the specific episodes experienced by the expert system.</li>
  <li>You might have expert systems that only work on a limited state space and do not scale to bigger, more complex environments. Therefore, although you <strong>can't</strong> use expert demonstrations in the bigger envs you still want to leverage their experience by learning their behavior in limited settings and the way to avoid overfitting to a specific, constrained policy is by always having some proportion of episodes generated by your policy being learned.</li>
  </ol>
  
Tags: <reinforcement-learning><machine-learning><deep-rl><imitation-learning>
Title: Why not use only expert demonstrations in Imitation Learning approaches?
UserDisplayName: Samuel Rodríguez
ThreadText: # Why not use only expert demonstrations in Imitation Learning approaches?
  
  (Asked by: username_0 on 2022-08-11)
  
  Some IL approaches train the agents by using some specific ratio of expert demonstrations to trajectories generated using the policy being optimized.
  
  In the specific [paper](https://arxiv.org/abs/1809.03531) I'm reading they say *"we experimented with various IL proportions (10-50% by increments of 10%) and observed that the RL/IL ratio does not seem to affect the performance of the trained policy by much."*
  
  My question is: why not rely *only* on expert demonstrations instead of introducing the noise of trajectories generated by a sub-optimal policy?.
  
  My assumptions are:
  
  1. This noise helps explore the state space beyond the specific episodes experienced by the expert system.
  2. You might have expert systems that only work on a limited state space and do not scale to bigger, more complex environments. Therefore, although you **can't** use expert demonstrations in the bigger envs you still want to leverage their experience by learning their behavior in limited settings and the way to avoid overfitting to a specific, constrained policy is by always having some proportion of episodes generated by your policy being learned.
  
  **Tags:** reinforcement-learning, machine-learning, deep-rl, imitation-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-29. Score: 0
  
  ## Using suboptimal demonstrations in Imitation Learning
  
  The paper you link above uses Behavior Cloning (supervised learning on the expert state-action pairs), which is known to suffer from the "DAgger problem" like many other imitation learning algorithms.
  
  In essence, a policy trained under imitation learning from expert demonstrations will inevitably make some suboptimal actions during the episode, and these suboptimal actions lead to states that are increasingly different from states in the expert demonstrations, which leads to more suboptimal actions.
  
  So there are cases where adding noise or perturbation to your demonstration set can lead to a more robust imitation policy[1].
  
  [1] T-Rex <https://arxiv.org/abs/1904.06387>
  
  ## Imitation Learning in PRIMAL
  
  As I understand it, the paper you link to only uses expert demonstrations during imitation learning. The suboptimal demonstrations are RL samples being used for an RL objective.
  
  The authors describe their choice to combine IL and RL as follows:
  
  > combining RL and IL has been shown to lead to faster, more stable
  > training as well as higher-quality solutions in robot manipulation.
  > These advantages are likely due to the fact that IL can help to
  > quickly identify high-quality regions of the agents state-action
  > space, while RL can further improve the policy by freely exploring
  > these regions
  
  
  ---
  
  