Answers.Body: <p>It's because, in the actor-critic algorithm, the objective function is an expectation under the <span class="math-container">$\tau$</span> of the policy. If we want to use off-policy data, we have to resort to <em>importance sampling</em> relative to the other policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-22T10:38:08.390Z
Answers.Text: What does the tau value mean?
Answers.UserDisplayName: asaf92
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-08T02:33:33.190Z
Answers.LastActivityDate: 2019-02-15T19:43:04.453Z
Answers.LastEditDate: 2019-02-15T19:43:04.453Z
Answers.UserDisplayName: apuffin
Body: <p>Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?</p>
  
Tags: <reinforcement-learning><actor-critic-methods><on-policy-methods><off-policy-methods>
Title: Why is the actor-critic algorithm limited to using on-policy data?
UserDisplayName: apuffin
ThreadText: # Why is the actor-critic algorithm limited to using on-policy data?
  
  (Asked by: username_0 on 2019-01-06)
  
  Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?
  
  **Tags:** reinforcement-learning, actor-critic-methods, on-policy-methods, off-policy-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-01-08. Score: 1
  
  It's because, in the actor-critic algorithm, the objective function is an expectation under the $\tau$ of the policy. If we want to use off-policy data, we have to resort to *importance sampling* relative to the other policy.
  
  > **Comments:**
  > **username_1**: What does the tau value mean?  
  
  ---
  
  