Answers.Body: <p>What you're looking for is called <em>Imitation Learning</em> (IL), in which we are interested in learning an expert policy <span class="math-container">$\pi_*$</span> which we assume to be <em>optimal</em>.</p>
  <p>However, there are many different ways we can approach such learning setting.
  Just to give some examples, we might be interested in <strong>Behavioural Cloning</strong>, where our <em>parametrised</em> policy <span class="math-container">$\pi_\theta$</span> (the RL agent) is trained in a supervised fashion to mimic the expert behaviour on a set of <em>demonstrations</em> <span class="math-container">$\mathcal{D} = \{(s_i, \pi_*(s_i))\}^N_{i=1}$</span>:</p>
  <p><span class="math-container">$\theta^* = argmin_\theta \sum_{s \in \mathcal{D}}\mathcal{L}(\pi_*(s),  \pi_\theta(s))$</span></p>
  <p>where <span class="math-container">$s_i$</span> represents a <em>state</em> and <span class="math-container">$\pi(s_i)$</span> the action taken by the policy.</p>
  <p>Other approaches involve learning a reward function <span class="math-container">$\mathcal{R}_\phi$</span> from <span class="math-container">$\mathcal{D}$</span> and use it to reward our agent (<strong>Inverse Reinforcement Learning</strong>) or GAIL (<strong>Generative Adversarial Imitation Learning</strong>). I suggest you take a look at <a href="https://smartlabai.medium.com/a-brief-overview-of-imitation-learning-8a8a75c44a9c" rel="nofollow noreferrer">this brief overview</a>.</p>
  <p>So, to answer your question: yes, you can leverage expert behaviour in your RL algorithm. <em>Should you do so?</em> It depends. IL is not always advisable as the agent might get stuck in a suboptimal policy, especially if the expert you're learning from is poor.</p>
  <p>You can also combine IL and RL by starting with the former early on the training and then switch to the latter for later epochs.</p>
  <p>Unfortunately, no way of knowing which works best a priori.</p>
  <p>Hope this was useful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-30T15:27:36.870Z
Answers.LastActivityDate: 2021-07-30T15:27:36.870Z
Answers.UserDisplayName: danin
Body: <p>I have a heuristic solution to a problem which works quite well when certain environmental parameters are known and unchanging. However, in a real world setting these parameters will not be known and are likely to fluctuate over the course of an episode. I'm hoping to use deep RL to develop a policy that will be similar to the heuristic, but robust to these unknowns.</p>
  <p>My question is: does the RL agent need to be trained &quot;from scratch&quot; as one would typically do or is there a way to leverage the existing policy to jump start the training progress?</p>
  <p>In the latter case, what would this looks like? I've had a couple of thoughts, but I'm not sure how well any of them would work.</p>
  <ol>
  <li><p>Reward actions that the heuristic would take in an environment with static parameter values, then gradually make the environment more complex and set a new reward function based on what I'm actually interested in.</p>
  </li>
  <li><p>Instead of taking random actions in the exploration stage, take actions dictated by the heuristic.</p>
  </li>
  </ol>
  
Tags: <reinforcement-learning><deep-rl><sample-efficiency>
Title: How to use a heuristic policy to increase sample efficiency of a deep reinforcement learning agent?
UserDisplayName: asheets
ThreadText: # How to use a heuristic policy to increase sample efficiency of a deep reinforcement learning agent?
  
  (Asked by: username_0 on 2021-07-29)
  
  I have a heuristic solution to a problem which works quite well when certain environmental parameters are known and unchanging. However, in a real world setting these parameters will not be known and are likely to fluctuate over the course of an episode. I'm hoping to use deep RL to develop a policy that will be similar to the heuristic, but robust to these unknowns.
  
  My question is: does the RL agent need to be trained "from scratch" as one would typically do or is there a way to leverage the existing policy to jump start the training progress?
  
  In the latter case, what would this looks like? I've had a couple of thoughts, but I'm not sure how well any of them would work.
  
  1. Reward actions that the heuristic would take in an environment with static parameter values, then gradually make the environment more complex and set a new reward function based on what I'm actually interested in.
  2. Instead of taking random actions in the exploration stage, take actions dictated by the heuristic.
  
  **Tags:** reinforcement-learning, deep-rl, sample-efficiency
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-30. Score: 2 | Accepted answer
  
  What you're looking for is called *Imitation Learning* (IL), in which we are interested in learning an expert policy $\pi\_\*$ which we assume to be *optimal*.
  
  However, there are many different ways we can approach such learning setting.
  Just to give some examples, we might be interested in **Behavioural Cloning**, where our *parametrised* policy $\pi\_\theta$ (the RL agent) is trained in a supervised fashion to mimic the expert behaviour on a set of *demonstrations* $\mathcal{D} = \{(s\_i, \pi\_\*(s\_i))\}^N\_{i=1}$:
  
  $\theta^\* = argmin\_\theta \sum\_{s \in \mathcal{D}}\mathcal{L}(\pi\_\*(s), \pi\_\theta(s))$
  
  where $s\_i$ represents a *state* and $\pi(s\_i)$ the action taken by the policy.
  
  Other approaches involve learning a reward function $\mathcal{R}\_\phi$ from $\mathcal{D}$ and use it to reward our agent (**Inverse Reinforcement Learning**) or GAIL (**Generative Adversarial Imitation Learning**). I suggest you take a look at [this brief overview](https://smartlabai.medium.com/a-brief-overview-of-imitation-learning-8a8a75c44a9c).
  
  So, to answer your question: yes, you can leverage expert behaviour in your RL algorithm. *Should you do so?* It depends. IL is not always advisable as the agent might get stuck in a suboptimal policy, especially if the expert you're learning from is poor.
  
  You can also combine IL and RL by starting with the former early on the training and then switch to the latter for later epochs.
  
  Unfortunately, no way of knowing which works best a priori.
  
  Hope this was useful.
  
  
  ---
  
  