Answers.Body: <p>You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy <span class="math-container">$\beta$</span> either.</p>
  <p>Essentially when they take the expectation with respect to <span class="math-container">$\rho^\beta$</span> they are taking expectation with respect to a state distribution induced by some policy <span class="math-container">$\beta$</span> that is not necessarily the same as our current policy -- this is what makes DDPG an off-policy algorithm.</p>
  <p>The target actor network <span class="math-container">$\mu'$</span> is used in the loss function for the critic; the target for the critic (ignoring parameters for brevity) <span class="math-container">$y = r + \gamma Q(s', \mu'(s'))$</span> where <span class="math-container">$s'$</span> is the state we transitioned to from <span class="math-container">$s$</span> when we took an action <span class="math-container">$a$</span>.</p>
  <p>Now, <span class="math-container">$a$</span> was sampled as part of the tuple <span class="math-container">$(s, a, r, s')$</span> from our replay buffer, meaning that <span class="math-container">$a$</span> will have been chosen according to some past version of our policy, plus some exploration noise <span class="math-container">$\mathcal{N}$</span>. The way this links to the behaviour policy <span class="math-container">$\beta$</span> is that because we have sampled it from some old version of our policy, i.e. not our current policy, it is instead coming from this behaviour policy <span class="math-container">$\beta$</span>.</p>
  <p>The target network <span class="math-container">$\mu'$</span> is simply a copy of the current actor network where the weights are updated using the polyak averaging technique and is not really related to the behaviour policy <span class="math-container">$\beta$</span>, at least not in any useful way for you to think about it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-16T17:28:33.663Z
Answers.LastActivityDate: 2021-03-16T17:28:33.663Z
Answers.UserDisplayName: David
Body: <p>I am learning about the <a href="https://arxiv.org/pdf/1509.02971.pdf" rel="nofollow noreferrer">deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016)</a> and got confused about the notation of the behavior policy.</p>
  <p><a href="https://arxiv.org/pdf/1509.02971.pdf" rel="nofollow noreferrer">Lillicrap et al.</a> denote the policy gradient by</p>
  <p><span class="math-container">$$\nabla _{\theta^\mu} J \approx \mathbb{E}_{s_t \sim \rho^\beta} \left[ \nabla _{\theta^\mu} Q(s,a|\theta^Q) | s=s_t, a=\mu(s_t ; \theta ^\mu)  \right],$$</span></p>
  <p>where <span class="math-container">$\beta$</span> denotes the behavior policy (equation 5 in the original paper).</p>
  <p>However, when they talk about exploration, they denote the exploration policy by <span class="math-container">$ \mu'$</span>. This notation seems confusing to me since the target actor network is also denoted by <span class="math-container">$\mu'(s|\theta^{\mu'})$</span>.</p>
  <p>As far as I understand, the exploration policy is not directly linked to the target critic network but rather corresponds to the previously mentioned behavior policy <span class="math-container">$\beta$</span>. Is this correct or am I understanding it wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-16T17:20:02.360Z
Comments.Text: I think you mean target actor network, not the target critic
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-18T10:03:24.907Z
Comments.Text: Did I answer your question? If not then please let me know and I will amend my answer based on your feedback, if so then please accept the question.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-18T12:25:09.370Z
Comments.Text: Thank you a lot for your explanations. Just to make sure I understand it correctly: The tuple $(s,a,r,s')$ was sampled with the (back then valid) exploration policy. However, the current exploration policy might have changed. Therefore, the authors have to introduce a new variable $\beta$ denoting "the policies, that were valid at the time back then".
Comments.UserDisplayName: Manuel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-18T12:27:47.460Z
Comments.Text: @DavidIreland Yes, I was referring to the actor network. I edited the original post. Thank you
Comments.UserDisplayName: Manuel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-18T14:41:17.633Z
Comments.Text: $\beta$ is not a variable, it is a policy. The introduction of $\beta$ means that the tuple $(s, a, r, s')$ does not need to be sampled according to our most recently policy, so we can sample previous experiences that come from the old policy as you say.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><deep-rl><policy-gradients><ddpg><notation>
Title: Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?
UserDisplayName: Manuel
ThreadText: # Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?
  
  (Asked by: username_0 on 2021-03-16)
  
  I am learning about the [deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016)](https://arxiv.org/pdf/1509.02971.pdf) and got confused about the notation of the behavior policy.
  
  [Lillicrap et al.](https://arxiv.org/pdf/1509.02971.pdf) denote the policy gradient by
  
  $$\nabla \_{\theta^\mu} J \approx \mathbb{E}\_{s\_t \sim \rho^\beta} \left[ \nabla \_{\theta^\mu} Q(s,a|\theta^Q) | s=s\_t, a=\mu(s\_t ; \theta ^\mu) \right],$$
  
  where $\beta$ denotes the behavior policy (equation 5 in the original paper).
  
  However, when they talk about exploration, they denote the exploration policy by $ \mu'$. This notation seems confusing to me since the target actor network is also denoted by $\mu'(s|\theta^{\mu'})$.
  
  As far as I understand, the exploration policy is not directly linked to the target critic network but rather corresponds to the previously mentioned behavior policy $\beta$. Is this correct or am I understanding it wrong?
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, ddpg, notation
  
  > **Comments:**
  > **username_1**: I think you mean target actor network, not the target critic  
  > **username_1**: Did I answer your question? If not then please let me know and I will amend my answer based on your feedback, if so then please accept the question.  
  > **username_0**: Thank you a lot for your explanations. Just to make sure I understand it correctly: The tuple $(s,a,r,s')$ was sampled with the (back then valid) exploration policy. However, the current exploration policy might have changed. Therefore, the authors have to introduce a new variable $\beta$ denoting "the policies, that were valid at the time back then".  
  > **username_0**: @DavidIreland Yes, I was referring to the actor network. I edited the original post. Thank you  
  > **username_1**: $\beta$ is not a variable, it is a policy. The introduction of $\beta$ means that the tuple $(s, a, r, s')$ does not need to be sampled according to our most recently policy, so we can sample previous experiences that come from the old policy as you say.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-16. Score: 1 | Accepted answer
  
  You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either.
  
  Essentially when they take the expectation with respect to $\rho^\beta$ they are taking expectation with respect to a state distribution induced by some policy $\beta$ that is not necessarily the same as our current policy -- this is what makes DDPG an off-policy algorithm.
  
  The target actor network $\mu'$ is used in the loss function for the critic; the target for the critic (ignoring parameters for brevity) $y = r + \gamma Q(s', \mu'(s'))$ where $s'$ is the state we transitioned to from $s$ when we took an action $a$.
  
  Now, $a$ was sampled as part of the tuple $(s, a, r, s')$ from our replay buffer, meaning that $a$ will have been chosen according to some past version of our policy, plus some exploration noise $\mathcal{N}$. The way this links to the behaviour policy $\beta$ is that because we have sampled it from some old version of our policy, i.e. not our current policy, it is instead coming from this behaviour policy $\beta$.
  
  The target network $\mu'$ is simply a copy of the current actor network where the weights are updated using the polyak averaging technique and is not really related to the behaviour policy $\beta$, at least not in any useful way for you to think about it.
  
  
  ---
  
  