Answers.Body: <p>The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the <em>same</em> and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different.</p>
  
  <p>The key to the understanding is that in on-policy algorithms you have to use actions (and generally speaking trajectories) generated by the policy in the updating steps (to improve the policy itself). This means that in the on-policy case $a_i = \mu(s_{i})$ (in equations 11-13).</p>
  
  <p>Whereas in the off-policy case you can use <em>any</em> trajectory to improve your value/action-value functions, which means that the actions $a_t$ can be generated by any distribution $a_t~\pi(s_t,a_t)$. In (16) the algorithm explicitly states however that the action-value function ($Q^w$) has to be evaluated at $\mu_{s_{t+1}}$ (just like in the on-policy case) and not at $a_t$ which was the actual action in the trajectory generated by policy $\pi$.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-09T13:22:29.027Z
Answers.LastActivityDate: 2018-05-09T13:22:29.027Z
Answers.UserDisplayName: Hai Nguyen
Answers.Body: <p>The main difference between on-policy and off-policy is how to get samples and what policy we optimize. </p>
  
  <p>In off-policy deterministic actor-critic, the trajectories are samples from beta distribution (also called behavior policy), not the policy we are optimized (that is <span class="math-container">$\mu_{\theta}$</span>). However, in the on-policy actor-critic, the action <span class="math-container">$a_{t+1}$</span> is sampled from target policy <span class="math-container">$\mu_{\theta}$</span> and the policy we optimized is also the <span class="math-container">$\mu_{\theta}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T01:42:38.157Z
Answers.LastActivityDate: 2019-04-24T09:29:01.420Z
Answers.LastEditDate: 2019-04-24T09:29:01.420Z
Answers.UserDisplayName: Jack Wang
Body: <p>In the paper <a href="http://proceedings.mlr.press/v32/silver14.pdf" rel="noreferrer"><em>Deterministic Policy Gradient Algorithms</a></em>, I am really confused about chapter 4.1 and 4.2 which is "On and off-policy Deterministic Actor-Critic". </p>
  
  <p>I don't know what's the difference between two algorithms. </p>
  
  <p>I only noticed that the equation 11 and 16 are different, and the difference is the action part of Q function where is $a_{t+1}$ in equation 11 and $\mu(s_{t+1})$ in equation 16. If that's what really matters, how can I calculate $a_{t+1}$ in equation 11?</p>
  
  <p><img src="https://s26.postimg.cc/y59xkd9q1/image.png" alt="">
  <img src="https://s26.postimg.cc/7wysv3zy1/image.png" alt=""></p>
  
Tags: <reinforcement-learning><terminology><actor-critic-methods><on-policy-methods><off-policy-methods>
Title: What is the difference between on and off-policy deterministic actor-critic?
UserDisplayName: fish_tree
ThreadText: # What is the difference between on and off-policy deterministic actor-critic?
  
  (Asked by: username_0 on 2018-05-09)
  
  In the paper [*Deterministic Policy Gradient Algorithms*](http://proceedings.mlr.press/v32/silver14.pdf), I am really confused about chapter 4.1 and 4.2 which is "On and off-policy Deterministic Actor-Critic".
  
  I don't know what's the difference between two algorithms.
  
  I only noticed that the equation 11 and 16 are different, and the difference is the action part of Q function where is $a\_{t+1}$ in equation 11 and $\mu(s\_{t+1})$ in equation 16. If that's what really matters, how can I calculate $a\_{t+1}$ in equation 11?
  
  ![](https://s26.postimg.cc/y59xkd9q1/image.png)
  ![](https://s26.postimg.cc/7wysv3zy1/image.png)
  
  **Tags:** reinforcement-learning, terminology, actor-critic-methods, on-policy-methods, off-policy-methods
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-05-09. Score: 4 | Accepted answer
  
  The twist here is that the $a\_{t+1}$ in (11) and the $\mu(s\_{t+1})$ in (16) are the *same* and actually the $a\_t$ in the on-policy case and the $a\_t$ in the off-policy case are different.
  
  The key to the understanding is that in on-policy algorithms you have to use actions (and generally speaking trajectories) generated by the policy in the updating steps (to improve the policy itself). This means that in the on-policy case $a\_i = \mu(s\_{i})$ (in equations 11-13).
  
  Whereas in the off-policy case you can use *any* trajectory to improve your value/action-value functions, which means that the actions $a\_t$ can be generated by any distribution $a\_t~\pi(s\_t,a\_t)$. In (16) the algorithm explicitly states however that the action-value function ($Q^w$) has to be evaluated at $\mu\_{s\_{t+1}}$ (just like in the on-policy case) and not at $a\_t$ which was the actual action in the trajectory generated by policy $\pi$.
  
  
  ---
  
  ### Answer by username_2 on 2019-04-24. Score: 1
  
  The main difference between on-policy and off-policy is how to get samples and what policy we optimize.
  
  In off-policy deterministic actor-critic, the trajectories are samples from beta distribution (also called behavior policy), not the policy we are optimized (that is $\mu\_{\theta}$). However, in the on-policy actor-critic, the action $a\_{t+1}$ is sampled from target policy $\mu\_{\theta}$ and the policy we optimized is also the $\mu\_{\theta}$.
  
  
  ---
  
  