Answers.Body: <p>No. An off-policy algorithm is, in one sense, always better. It allows you to train on already collected data that you could otherwise not use.</p>
  <p>An off-policy algorithm can be used to &quot;re-interpret&quot; data that was collected earlier while running a worse policy. It can also learn the optimal policy while still doing disruptive random exploration. (Instead of learning to anticipate those random actions and mitigate.)</p>
  <p>But in practice, if you can cheaply generate new training episodes (e.g. with a fast simulator), an off-policy algorithm may make the learning more complicated (slower) without any benefit. As always, it depends. Every method has some task for which it is optimal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-02T12:21:22.073Z
Answers.LastActivityDate: 2023-04-02T12:40:08.073Z
Answers.LastEditDate: 2023-04-02T12:40:08.073Z
Answers.UserDisplayName: maxy
Body: <p>I am studying RL and I have a question: Are <strong>on-policy</strong> algorithms always better than <strong>off-policy</strong> ones?</p>
  
Tags: <reinforcement-learning><comparison><off-policy-methods><on-policy-methods>
Title: Are on-policy algorithms always better than off-policy ones?
UserDisplayName: Samvel Safaryan
ThreadText: # Are on-policy algorithms always better than off-policy ones?
  
  (Asked by: username_0 on 2023-04-02)
  
  I am studying RL and I have a question: Are **on-policy** algorithms always better than **off-policy** ones?
  
  **Tags:** reinforcement-learning, comparison, off-policy-methods, on-policy-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-02. Score: 0 | Accepted answer
  
  No. An off-policy algorithm is, in one sense, always better. It allows you to train on already collected data that you could otherwise not use.
  
  An off-policy algorithm can be used to "re-interpret" data that was collected earlier while running a worse policy. It can also learn the optimal policy while still doing disruptive random exploration. (Instead of learning to anticipate those random actions and mitigate.)
  
  But in practice, if you can cheaply generate new training episodes (e.g. with a fast simulator), an off-policy algorithm may make the learning more complicated (slower) without any benefit. As always, it depends. Every method has some task for which it is optimal.
  
  
  ---
  
  