Answers.Body: <p>Your second equation is defined on page 78 and in the same page there's a step on the lower part of derivations contains the answer to your confusion.</p>
  <blockquote>
  <p><span class="math-container">$q_{\pi}(s, {\pi'}(s)) \\= E[R_{t+1} + \gamma v_{\pi}(S_{t+1})|S_t=s,A_t={\pi'}(s)] \\= E_{\pi'}[R_{t+1}+ \gamma v_{\pi}(S_{t+1})|S_t=s]$</span></p>
  </blockquote>
  <p>You see the key is when your expectation is conditioned on a specific action at time <span class="math-container">$t$</span>, the only remaining random variable as the capitalized reward <span class="math-container">$R_{t+1}$</span> is solely determined by the stochastic environment not any agent's policy. If it's not conditioned on a specific action at time <span class="math-container">$t$</span>, then according to the backup diagram in your another post the only remaining random variable as the capitalized reward <span class="math-container">$R_{t+1}$</span> is <em>not</em> solely determined by the stochastic environment but also determined by which specific action <span class="math-container">$a$</span> is taken at time <span class="math-container">$t$</span> probabilistically based on the distribution of the random action <span class="math-container">$A_t$</span> controlled by the policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-16T18:00:32.147Z
Answers.LastActivityDate: 2024-02-16T18:00:32.147Z
Answers.UserDisplayName: cinch
Answers.Body: <p>The subscript just indicates over which variable the expectation is taken. For consistency it can be included everywhere. Sometimes, when the variable over which the expectation is taken is obvious, it is not precised. For example <span class="math-container">$E[x]$</span>. In that case it's clear, however you can imagine, that the variable x is dependent on some latent feature and the expectation is not over the variable x, but its latent dependence, e.g. <span class="math-container">$E_z[x(z)] = E_z[x]$</span>.</p>
  <p>Regarding your case, I would assume, that in first case its emphasised that the expectation is taken over all the actions following the policy <span class="math-container">$\pi$</span>. In the second case, you can conclude from the context, what is the variable of interest is.</p>
  <p>You can find very similar question <a href="https://ai.stackexchange.com/questions/36087/e-pir-t1s-t-s-a-t-a-er-t1s-t-s-a-t-a?rq=1">here</a>. Hope it helps!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-16T17:14:32.227Z
Answers.Text: I think reasoning is different.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-16T17:19:20.407Z
Answers.Text: Following the answers, from the question I linked It may be that the  q is not dependent on the policy, it will appear strange to me however, since you put a subscript at q in your question. Can you share the reference to the equation, that is creating the confusion?
Answers.UserDisplayName: vl_knd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-16T17:26:08.993Z
Answers.Text: I think it is used to mean "conditioning" on $\pi$, but I want to understand this logic fully. These equations are from the solutions manual to an exercise, hence I cannot share.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-16T17:43:12.273Z
Answers.Text: You can say, that "conditioning" on $\pi$ in first case can mean that your actions are distributed with density $\pi$, so it will appear in expectation calculation. 
  
  However, it looks to me, that in the second expectation, the action is fixed by conditional expectation $E[\cdot | S_t = s, A_t = a]$. So the variable of interest for expectation is something else. You probably can conclude from context which variable it is.
Answers.UserDisplayName: vl_knd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-16T17:12:12.907Z
Answers.LastActivityDate: 2024-02-16T17:39:06.223Z
Answers.LastEditDate: 2024-02-16T17:39:06.223Z
Answers.UserDisplayName: vl_knd
Body: <p>Here are two relations for value functions:</p>
  <p>Eq.1: <span class="math-container">$v_{\pi}(s)=E_{\pi}[q_{\pi}(S_t, A_t)|S_t=s]$</span></p>
  <p>Eq.1: <span class="math-container">$q_{\pi}(s,a)=E[R_{t+1}+\gamma v_{\pi}(S_{t+1})|S_t=s, A_t=a]$</span></p>
  <p>Question: Why is there subscript <span class="math-container">$\pi$</span> in the first expectation equation, but not in the second one? What is the logic in including/not including subscript <span class="math-container">$\pi$</span> in &quot;E&quot;?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-19T12:07:32.503Z
Comments.Text: Where did you find these equations?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><markov-decision-process>
Title: What is the logic in including/not including subscript $\pi$ in in "E" for value functions?
UserDisplayName: DSPinfinity
ThreadText: # What is the logic in including/not including subscript $\pi$ in in "E" for value functions?
  
  (Asked by: username_0 on 2024-02-16)
  
  Here are two relations for value functions:
  
  Eq.1: $v\_{\pi}(s)=E\_{\pi}[q\_{\pi}(S\_t, A\_t)|S\_t=s]$
  
  Eq.1: $q\_{\pi}(s,a)=E[R\_{t+1}+\gamma v\_{\pi}(S\_{t+1})|S\_t=s, A\_t=a]$
  
  Question: Why is there subscript $\pi$ in the first expectation equation, but not in the second one? What is the logic in including/not including subscript $\pi$ in "E"?
  
  **Tags:** reinforcement-learning, markov-decision-process
  > **Comments:**
  > **username_1**: Where did you find these equations?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-02-16. Score: 1 | Accepted answer
  
  Your second equation is defined on page 78 and in the same page there's a step on the lower part of derivations contains the answer to your confusion.
  
  > $q\_{\pi}(s, {\pi'}(s)) \\= E[R\_{t+1} + \gamma v\_{\pi}(S\_{t+1})|S\_t=s,A\_t={\pi'}(s)] \\= E\_{\pi'}[R\_{t+1}+ \gamma v\_{\pi}(S\_{t+1})|S\_t=s]$
  
  You see the key is when your expectation is conditioned on a specific action at time $t$, the only remaining random variable as the capitalized reward $R\_{t+1}$ is solely determined by the stochastic environment not any agent's policy. If it's not conditioned on a specific action at time $t$, then according to the backup diagram in your another post the only remaining random variable as the capitalized reward $R\_{t+1}$ is *not* solely determined by the stochastic environment but also determined by which specific action $a$ is taken at time $t$ probabilistically based on the distribution of the random action $A\_t$ controlled by the policy.
  
  
  ---
  
  ### Answer by username_3 on 2024-02-16. Score: 1
  
  The subscript just indicates over which variable the expectation is taken. For consistency it can be included everywhere. Sometimes, when the variable over which the expectation is taken is obvious, it is not precised. For example $E[x]$. In that case it's clear, however you can imagine, that the variable x is dependent on some latent feature and the expectation is not over the variable x, but its latent dependence, e.g. $E\_z[x(z)] = E\_z[x]$.
  
  Regarding your case, I would assume, that in first case its emphasised that the expectation is taken over all the actions following the policy $\pi$. In the second case, you can conclude from the context, what is the variable of interest is.
  
  You can find very similar question [here](https://ai.stackexchange.com/questions/36087/e-pir-t1s-t-s-a-t-a-er-t1s-t-s-a-t-a?rq=1). Hope it helps!
  
  > **Comments:**
  > **username_0**: I think reasoning is different.  
  > **username_3**: Following the answers, from the question I linked It may be that the  q is not dependent on the policy, it will appear strange to me however, since you put a subscript at q in your question. Can you share the reference to the equation, that is creating the confusion?  
  > **username_0**: I think it is used to mean "conditioning" on $\pi$, but I want to understand this logic fully. These equations are from the solutions manual to an exercise, hence I cannot share.  
  > **username_3**: You can say, that "conditioning" on $\pi$ in first case can mean that your actions are distributed with density $\pi$, so it will appear in expectation calculation.   However, it looks to me, that in the second expectation, the action is fixed by conditional expectation $E[\cdot | S_t = s, A_t = a]$. So the variable of interest for expectation is something else. You probably can conclude from context which variable it is.  
  
  ---
  
  