Answers.Body: <p>SAC is an off-policy method because it learns from a replay buffer, which contains experiences collected by the agent over time from potentially different versions of the policy. This means the agent can learn from past experiences generated by older versions of the policy, which might not be the same as the current policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-21T08:11:02.360Z
Answers.LastActivityDate: 2023-03-21T08:11:02.360Z
Answers.UserDisplayName: Hans-Peter Schrei
Body: <p>I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like <span class="math-container">$\epsilon$</span>-greedy technique for exploration while training our DNN, so this makes the behavior policy (<span class="math-container">$\epsilon$</span>-greedy) and the target policy (from DNN) to be different.</p>
  <p>However, in the case of soft actor-critic (SAC) we use the same policy DNN to generate the data stored in the memory buffer. Later we use this data to update the policy DNN. There is only one DNN that is being used for generating the data for exploration, while this data is used for updating the policy network (same DNN). When the DNN is the same, how can we say that the behavior policy and target policy are different?</p>
  
Tags: <reinforcement-learning><actor-critic-methods><soft-actor-critic>
Title: Why is soft actor critic an off policy scheme?
UserDisplayName: ZZ1
ThreadText: # Why is soft actor critic an off policy scheme?
  
  (Asked by: username_0 on 2023-03-21)
  
  I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like $\epsilon$-greedy technique for exploration while training our DNN, so this makes the behavior policy ($\epsilon$-greedy) and the target policy (from DNN) to be different.
  
  However, in the case of soft actor-critic (SAC) we use the same policy DNN to generate the data stored in the memory buffer. Later we use this data to update the policy DNN. There is only one DNN that is being used for generating the data for exploration, while this data is used for updating the policy network (same DNN). When the DNN is the same, how can we say that the behavior policy and target policy are different?
  
  **Tags:** reinforcement-learning, actor-critic-methods, soft-actor-critic
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-21. Score: 3 | Accepted answer
  
  SAC is an off-policy method because it learns from a replay buffer, which contains experiences collected by the agent over time from potentially different versions of the policy. This means the agent can learn from past experiences generated by older versions of the policy, which might not be the same as the current policy.
  
  
  ---
  
  