Body: <p>I am currently learning policy gradient methods from the <a href="https://www.youtube.com/watch?v=S_gwYj1Q-44&amp;t=1241s" rel="nofollow noreferrer">Deep RL boot camp by Pieter Abbeel</a> in which he explains the actor-critic algorithm derivation. </p>
  
  <p>At around minute 39, he explains that the sum of the rewards from time step <span class="math-container">$t$</span> onwards is actually an estimation of <span class="math-container">$Q^\pi(s,u)$</span>. I understand the definition of <span class="math-container">$Q^\pi(s,u)$</span> but I'm not sure why this is the case here. Is the reward following after time step <span class="math-container">$t+1$</span> collected based on current policy?</p>
  
  <p><a href="https://i.sstatic.net/8DOfH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8DOfH.png" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-10T17:40:19.170Z
Comments.Text: yes, you are getting rewards following the policy $\pi$. After completing each trajectory, then you will update the policy $\pi$.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-11T02:22:07.447Z
Comments.Text: so at state $s_k^(i)$ the action taken is $u_k$ but for state $s_{k+1}$ onwards, the policy $\pi$ is being followed ? but the notation does not contain anything about $\pi$ in the $R(s_k,u_k)$ term
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-11T05:39:27.047Z
Comments.Text: The notation $R(s_{k},u_{k})$ is not clear, but the rewards you get is following your current policy
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-11T07:03:07.907Z
Comments.Text: but why would that be an estimation of Q ? if following the current policy it should be $V^\pi$ ?
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-11T07:53:36.073Z
Comments.Text: You are sampling action($u_{k}$) from the current policy, $u_{k} \sim \pi$. So, it is like that, you are at the state $s_{k}$, from that state sample action $u_{k}$ from current policy $\pi$, thereafter following the policy $\pi$. By defination this is action value function ($Q^{\pi}(s_{k},u_{k})$). Don't be confused with:   $Q^{\pi}(s_{t},\pi(a_{t}|s_{t})$ and $Q^{\pi}(s,a)$ where $a \sim \pi$, those two have different meaning.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-11T13:11:54.563Z
Comments.Text: hmm how is sampling action from the current policy different from following the policy ? because the policy $\pi_{\theta}$ is a distribution of actions and $u_k$ is an action that is sampled from this policy distribution ?
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-11T15:20:21.667Z
Comments.Text: Consider that you are following  a stochastic policy since this policy gradient algorithm gives the stochastic policy as output. So, $Q^{\pi}(s_{t},\pi(a_{t}|s_{t})) = \sum_{a} \pi(a|s) \sum_{s',r} p(s',r|s,a) (r+\gamma v^{\pi}(s') = V^{\pi}(s)$ and $Q^{\pi}(s_{t},a) = \sum_{s',r} p(s',r|s,a) (r+\gamma v^{\pi}(s'))$ where $a \sim \pi$
Comments.UserDisplayName: Swakshar Deb
Tags: <reinforcement-learning><policy-gradients><actor-critic-methods>
Title: Is the reward following after time step $t+1$ collected based on current policy?
UserDisplayName: calveeen
ThreadText: # Is the reward following after time step $t+1$ collected based on current policy?
  
  (Asked by: username_0 on 2020-05-10)
  
  I am currently learning policy gradient methods from the [Deep RL boot camp by Pieter Abbeel](https://www.youtube.com/watch?v=S_gwYj1Q-44&t=1241s) in which he explains the actor-critic algorithm derivation.
  
  At around minute 39, he explains that the sum of the rewards from time step $t$ onwards is actually an estimation of $Q^\pi(s,u)$. I understand the definition of $Q^\pi(s,u)$ but I'm not sure why this is the case here. Is the reward following after time step $t+1$ collected based on current policy?
  
  [![enter image description here](https://i.sstatic.net/8DOfH.png)](https://i.sstatic.net/8DOfH.png)
  
  **Tags:** reinforcement-learning, policy-gradients, actor-critic-methods
  > **Comments:**
  > **username_1**: yes, you are getting rewards following the policy $\pi$. After completing each trajectory, then you will update the policy $\pi$.  
  > **username_0**: so at state $s_k^(i)$ the action taken is $u_k$ but for state $s_{k+1}$ onwards, the policy $\pi$ is being followed ? but the notation does not contain anything about $\pi$ in the $R(s_k,u_k)$ term  
  > **username_1**: The notation $R(s_{k},u_{k})$ is not clear, but the rewards you get is following your current policy  
  > **username_0**: but why would that be an estimation of Q ? if following the current policy it should be $V^\pi$ ?  
  > **username_1**: You are sampling action($u_{k}$) from the current policy, $u_{k} \sim \pi$. So, it is like that, you are at the state $s_{k}$, from that state sample action $u_{k}$ from current policy $\pi$, thereafter following the policy $\pi$. By defination this is action value function ($Q^{\pi}(s_{k},u_{k})$). Don't be confused with:   $Q^{\pi}(s_{t},\pi(a_{t}|s_{t})$ and $Q^{\pi}(s,a)$ where $a \sim \pi$, those two have different meaning.  
  > **username_0**: hmm how is sampling action from the current policy different from following the policy ? because the policy $\pi_{\theta}$ is a distribution of actions and $u_k$ is an action that is sampled from this policy distribution ?  
  > **username_1**: Consider that you are following  a stochastic policy since this policy gradient algorithm gives the stochastic policy as output. So, $Q^{\pi}(s_{t},\pi(a_{t}|s_{t})) = \sum_{a} \pi(a|s) \sum_{s',r} p(s',r|s,a) (r+\gamma v^{\pi}(s') = V^{\pi}(s)$ and $Q^{\pi}(s_{t},a) = \sum_{s',r} p(s',r|s,a) (r+\gamma v^{\pi}(s'))$ where $a \sim \pi$  
  
  ---
  
  ## 0 answers
  
  