Answers.Body: <p>What you're describing is off-policy learning. A classic example is <span class="math-container">$Q$</span>-learning, where you follow some policy <span class="math-container">$\pi$</span> whilst learning about the greedy policy.</p>
  <p>If you're interested in actor-critic methods then a popular off-policy method is the <a href="https://arxiv.org/pdf/1509.02971.pdf?source=post_page---------------------------" rel="nofollow noreferrer">Deep Deterministic Policy Gradient</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-17T13:46:51.447Z
Answers.LastActivityDate: 2020-08-17T13:46:51.447Z
Answers.UserDisplayName: David
Body: <p>When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a bunch of episodes using randomly sampled actions (or maybe by following an old policy) one time, and train multiple policies using this set of samples to find the most effective hyperparameters? Every time we train a new policy, does it make sense to replay all the episodes generated by the previous policy? I'm mostly interested in actor-critic methods.</p>
  
Tags: <reinforcement-learning><actor-critic-methods><off-policy-methods>
Title: Learning only using off-policy samples
UserDisplayName: smorad
ThreadText: # Learning only using off-policy samples
  
  (Asked by: username_0 on 2020-08-17)
  
  When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a bunch of episodes using randomly sampled actions (or maybe by following an old policy) one time, and train multiple policies using this set of samples to find the most effective hyperparameters? Every time we train a new policy, does it make sense to replay all the episodes generated by the previous policy? I'm mostly interested in actor-critic methods.
  
  **Tags:** reinforcement-learning, actor-critic-methods, off-policy-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-17. Score: 1
  
  What you're describing is off-policy learning. A classic example is $Q$-learning, where you follow some policy $\pi$ whilst learning about the greedy policy.
  
  If you're interested in actor-critic methods then a popular off-policy method is the [Deep Deterministic Policy Gradient](https://arxiv.org/pdf/1509.02971.pdf?source=post_page---------------------------).
  
  
  ---
  
  