Answers.Body: <p>I am not sure if it is standard notation, but Sutton &amp; Barto use a convention that a function of a random variable is a new random variable that maps between values of the old variable to values of the new one using the function, and without affecting probability distribution (other than the function could be one-way hence probabilties may effectively combine e.g. if there were several states with <span class="math-container">$v_*(s) = 5$</span>).</p>
  <p>Given this convention then <span class="math-container">$v_*(S_{t+1})$</span> is a random variable of the optimal state value functions of possible statues at time step <span class="math-container">$t+1$</span>. That is, it has the same probability densities, based on policy and state transition rules, as <span class="math-container">$S_{t+1}$</span>, but has the associated values of <span class="math-container">$v_*(s)$</span> for each possible <span class="math-container">$S_{t+1}$</span>.</p>
  <p>The actual distribution of <span class="math-container">$v_{*}(S_{t+1})$</span> will vary depending on the conditions in the context where it is evaluated.</p>
  <p>If you resolve the expectations in the first equation, which has conditions on <span class="math-container">$S_t$</span> and <span class="math-container">$A_t$</span>:</p>
  <p><span class="math-container">$q_{*}(s,a)=\mathbb{E}[R_{t+1}+\gamma v_{*}(S_{t+1})|S_{t}=s, A_{t}=a]$</span></p>
  <p><span class="math-container">$\qquad\quad= \sum_{r,s'} p(r,s'|s,a)(r + \gamma v_*(s'))$</span></p>
  <p>. . . which expresses action value <span class="math-container">$q_*(s,a)$</span> in terms of the state transition rules, immediate reward function and the state value <span class="math-container">$v_*(s')$</span> one half-step ahead. That is, at the next state, but before the next (optimal choice) action is taken.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-14T20:55:04.320Z
Answers.Text: I'm not sure if I understood well so let me please ask a question in general terms. If $X$ is a random variable from $\Omega$ to $\mathbb{R}$ and $f$ is a function on $\mathbb{R}$ then according to the convention, $f(X)$ is a random variable from $\Omega$ to $\mathbb{R}$ and the values of this random variable are $f(x)$ for $x$ in $\Omega$ but $P[f(X)=f(x)]=P[X=x]$ right ?
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-14T20:57:07.547Z
Answers.Text: @Daviiid Yes I think that has correctly put my first two paragraphs into notation. Sort of, I think you have some typos, i.e. $X$ is a random variable in $\Omega$ (the "to $\mathbb{R}$" after that looks like a typo), and $f(x): \Omega \rightarrow \mathbb{R}$
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-14T21:01:12.633Z
Answers.Text: Thank you for your answer. Can I ask please if there is a particular reason for going with this convention ? I think we can still get the same formula with the normal association of functions and random variables; $\mathbb{E}[v_{*}(S_{t+1})|S_{t}=s,A_{t}=a]=\sum_{s'}{p(s'|s,a)v_{*}(s')}=\sum_{r,s'}{p(r,s'|s,a)v_{*}(s')}$ since $\sum_{r}{p(r,s'|s,a)}=1$, or am I missing something ?
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-14T21:06:05.413Z
Answers.Text: @Daviiid I cannot see any difference between what you have written and I have done in the question, in terms of needing to interpret $v_*(S_{t+1})$. You have just resolved the expectation to the sum more conventionally (the double-variable sum $\sum_{r,s'}$ is a different conventon I have borrowed from Sutton & Barto). But I don't think it uses a different interpretation of what $v_*(S_{t+1})$ is? Sorry I am not a notation expert, and may have missed some subtlety.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-14T21:43:46.247Z
Answers.Text: I'd like to apologize first for not being able to add your name with the @. There is no difference between what I've written and your answer I just wanted to rewrite it as a clarification to my question. I just wanted to know if there is a particular reason for the authors for choosing their convention for functions of random variables. Because normally a function of a random variable would change the distribution of probabilities, we just imagine a scalar multiplication of a Gaussian random variable and how the distribution changes.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-15T06:50:50.987Z
Answers.Text: @Daviiid: OK, I think I understand the concern. With discrete probability distributions then the problems with the convention are minor and do not impact any of the sums. For continuous distributions you have to be more careful, although I think most of the manipulations of the expectations etc in RL would still be fine.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-15T07:58:24.517Z
Answers.Text: I understand now. Thank you for the explanations and all your clarifications.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-14T20:43:26.717Z
Answers.LastActivityDate: 2021-06-14T20:48:30.703Z
Answers.LastEditDate: 2021-06-14T20:48:30.703Z
Answers.UserDisplayName: Neil Slater
Body: <p>In Sutton &amp; Barto's <a href="http://incompleteideas.net/book/the-book.html" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> page 63 the authors introduce the optimal state value function in the expression of the optimal action-value function as follows: <span class="math-container">$q_{*}(s,a)=\mathbb{E}[R_{t+1}+\gamma v_{*}(S_{t+1})|S_{t}=s, A_{t}=a], \forall s \in S, \forall a \in A$</span>.</p>
  <p>I don't understand what <span class="math-container">$v_{*}(S_{t+1})$</span> could possibly mean since <span class="math-container">$v_{*}$</span> is a mapping, under the optimal policy <span class="math-container">$\pi_{*}$</span>, from states to numbers which are expected returns starting from those states and at different time steps.</p>
  <p>I believe that the authors use the same notation to denote the state-value function <span class="math-container">$v$</span> that verify <span class="math-container">$v(s)=\mathbb{E}[G_{t}|S_{t}=s], \forall s \in S$</span> and the random variable <span class="math-container">$\mathbb{E}[G_{t+1}|S_{t+1}]$</span> but I'm not sure.</p>
  
Tags: <reinforcement-learning><optimal-policy>
Title: What does $v(S_{t+1})$ mean in the optimal state-action value function?
UserDisplayName: Daviiid
ThreadText: # What does $v(S_{t+1})$ mean in the optimal state-action value function?
  
  (Asked by: username_0 on 2021-06-14)
  
  In Sutton & Barto's [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/the-book.html) page 63 the authors introduce the optimal state value function in the expression of the optimal action-value function as follows: $q\_{\*}(s,a)=\mathbb{E}[R\_{t+1}+\gamma v\_{\*}(S\_{t+1})|S\_{t}=s, A\_{t}=a], \forall s \in S, \forall a \in A$.
  
  I don't understand what $v\_{\*}(S\_{t+1})$ could possibly mean since $v\_{\*}$ is a mapping, under the optimal policy $\pi\_{\*}$, from states to numbers which are expected returns starting from those states and at different time steps.
  
  I believe that the authors use the same notation to denote the state-value function $v$ that verify $v(s)=\mathbb{E}[G\_{t}|S\_{t}=s], \forall s \in S$ and the random variable $\mathbb{E}[G\_{t+1}|S\_{t+1}]$ but I'm not sure.
  
  **Tags:** reinforcement-learning, optimal-policy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-14. Score: 2 | Accepted answer
  
  I am not sure if it is standard notation, but Sutton & Barto use a convention that a function of a random variable is a new random variable that maps between values of the old variable to values of the new one using the function, and without affecting probability distribution (other than the function could be one-way hence probabilties may effectively combine e.g. if there were several states with $v\_\*(s) = 5$).
  
  Given this convention then $v\_\*(S\_{t+1})$ is a random variable of the optimal state value functions of possible statues at time step $t+1$. That is, it has the same probability densities, based on policy and state transition rules, as $S\_{t+1}$, but has the associated values of $v\_\*(s)$ for each possible $S\_{t+1}$.
  
  The actual distribution of $v\_{\*}(S\_{t+1})$ will vary depending on the conditions in the context where it is evaluated.
  
  If you resolve the expectations in the first equation, which has conditions on $S\_t$ and $A\_t$:
  
  $q\_{\*}(s,a)=\mathbb{E}[R\_{t+1}+\gamma v\_{\*}(S\_{t+1})|S\_{t}=s, A\_{t}=a]$
  
  $\qquad\quad= \sum\_{r,s'} p(r,s'|s,a)(r + \gamma v\_\*(s'))$
  
  . . . which expresses action value $q\_\*(s,a)$ in terms of the state transition rules, immediate reward function and the state value $v\_\*(s')$ one half-step ahead. That is, at the next state, but before the next (optimal choice) action is taken.
  
  > **Comments:**
  > **username_0**: I'm not sure if I understood well so let me please ask a question in general terms. If $X$ is a random variable from $\Omega$ to $\mathbb{R}$ and $f$ is a function on $\mathbb{R}$ then according to the convention, $f(X)$ is a random variable from $\Omega$ to $\mathbb{R}$ and the values of this random variable are $f(x)$ for $x$ in $\Omega$ but $P[f(X)=f(x)]=P[X=x]$ right ?  
  > **username_1**: @Daviiid Yes I think that has correctly put my first two paragraphs into notation. Sort of, I think you have some typos, i.e. $X$ is a random variable in $\Omega$ (the "to $\mathbb{R}$" after that looks like a typo), and $f(x): \Omega \rightarrow \mathbb{R}$  
  > **username_0**: Thank you for your answer. Can I ask please if there is a particular reason for going with this convention ? I think we can still get the same formula with the normal association of functions and random variables; $\mathbb{E}[v_{*}(S_{t+1})|S_{t}=s,A_{t}=a]=\sum_{s'}{p(s'|s,a)v_{*}(s')}=\sum_{r,s'}{p(r,s'|s,a)v_{*}(s')}$ since $\sum_{r}{p(r,s'|s,a)}=1$, or am I missing something ?  
  > **username_1**: @Daviiid I cannot see any difference between what you have written and I have done in the question, in terms of needing to interpret $v_*(S_{t+1})$. You have just resolved the expectation to the sum more conventionally (the double-variable sum $\sum_{r,s'}$ is a different conventon I have borrowed from Sutton & Barto). But I don't think it uses a different interpretation of what $v_*(S_{t+1})$ is? Sorry I am not a notation expert, and may have missed some subtlety.  
  > **username_0**: I'd like to apologize first for not being able to add your name with the @. There is no difference between what I've written and your answer I just wanted to rewrite it as a clarification to my question. I just wanted to know if there is a particular reason for the authors for choosing their convention for functions of random variables. Because normally a function of a random variable would change the distribution of probabilities, we just imagine a scalar multiplication of a Gaussian random variable and how the distribution changes.  
  > **username_1**: @Daviiid: OK, I think I understand the concern. With discrete probability distributions then the problems with the convention are minor and do not impact any of the sums. For continuous distributions you have to be more careful, although I think most of the manipulations of the expectations etc in RL would still be fine.  
  > **username_0**: I understand now. Thank you for the explanations and all your clarifications.  
  
  ---
  
  