Answers.Body: <p>I am using the convention of uppercase <span class="math-container">$X$</span> for random variable and lowercase <span class="math-container">$x$</span> for an individual observation. It is possible your source material did not do this, which might be causing your confusion. However, it is the convention used in Sutton &amp; Barto's <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a>.</p>
  <blockquote>
  <p>What I didn't understand is what is  here. i.e., what is the random variable on which we are calculating long-run rewards?</p>
  </blockquote>
  <p>The random variable is <span class="math-container">$R_t$</span>, the reward at each time step. The distribution of <span class="math-container">$R_t$</span> in turn depends on the distribution of <span class="math-container">$S_{t-1}$</span> and <span class="math-container">$A_{t-1}$</span> plus the policy and state progression rules. There is no need to include the <em>process</em> that causes the distribution of each <span class="math-container">$R_t$</span> in every equation. Although sometimes it is useful to do so, for example when deriving the Bellman equations for value functions.</p>
  <blockquote>
  <p>My guess is policy function. It is averaging long-run rewards over all possible policy functions. Is it true?</p>
  </blockquote>
  <p>No, this is not true. In fact, it is the more usual assumption that the policy function <span class="math-container">$\pi(a|s)$</span> remains constant over the expectation, and this is what the subscript <span class="math-container">$\pi$</span> in <span class="math-container">$\mathbb{E}_{\pi}[...]$</span> means.</p>
  <p>The expectation is over randomness due to the policy <span class="math-container">$\pi$</span>, plus randomness due to the environment, which can be described by the function <span class="math-container">$p(r, s'|s, a)$</span> - the probability of observing reward <span class="math-container">$r$</span> and next state <span class="math-container">$s'$</span> given starting in state <span class="math-container">$s$</span> and taking action <span class="math-container">$a$</span>. These two functions combine to create the distribution of <span class="math-container">$R_t$</span>. It is possible that both functions are deterministic in practice, thus <span class="math-container">$R_t$</span> is also deterministic. However, RL theory works on the more general stochastic case, which is also used to model exploratory actions, even if the target policy and environment are deterministic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T10:10:52.867Z
Answers.Text: To me, it's not fully clear what you mean by this sentence "_There is no need to confuse the process that causes the distribution of each $R_t$ (which can be broken down analyitically, and this is done to derive the Bellman equations), with the fact that you are measuring $R_t$ directly in the expectation._"
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T11:24:08.370Z
Answers.Text: @nbro I've had an attempt to reword that paragraph.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-19T07:51:34.937Z
Answers.Text: If $R$ is the random variable, then all the possible values of $R, (r_{t+1}, r_{t+2}.....)$ are used in the summation. Then on which values does the expectation  is calculated on? I tried a lot, but can't able to figure out where I am going wrong. Is it on all possible reward functions?
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-19T08:05:17.740Z
Answers.Text: @hanugm If $R_t$ is a random variable, it has a _distribution_, The expectation is calculated over that distribution. The precise distribution depends on the policy being used, plus the state progression and reward function from the environment. Typically in RL you don't work with the full distribution of the return ($G_t$ or sum of rewards), although that may be possible for very simple environments. Instead you work either with samples from the distribution (e.g. Monte Carlo methods) or the Bellman equation (e.g. Value Iteration) or both (e.g. TD learning).
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-26T03:02:52.813Z
Answers.Text: So, If I have a fixed Immediate reward function and policy function, then there is no need of the expectation. Is it right?
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-26T08:44:59.653Z
Answers.Text: @hanugm:  You would need a fixed *deterministic* policy, a *deterministic* reward function and a *deterministic* state progression. Then in theory all sampled results would exactly equal the expectation. That is not a usual setup for RL, because you also want to explore and typically that is done by having a stochastic policy.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-26T08:49:52.450Z
Answers.Text: @hanugm: The equations with the expectation are used to figure out RL theory.  They are directly relevant for methods like Value Iteration. Other than that, unless you are working on the thoery (studying existing approaches, or devising new ones) then you don't need to work with it directly.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-17T10:00:35.103Z
Answers.LastActivityDate: 2022-01-14T23:51:23.303Z
Answers.LastEditDate: 2022-01-14T23:51:23.303Z
Answers.UserDisplayName: Neil Slater
Body: <p>I know that</p>
  <p><span class="math-container">$$\mathbb{E}[g(X) \mid A] = \sum\limits_{x} g(x) p_{X \mid A}(x)$$</span></p>
  <p>for any random variable <span class="math-container">$X$</span>.</p>
  <p>Now, consider the following expression.</p>
  <p><span class="math-container">$$\mathbb{E}_{\pi} \left[ \sum \limits_{k=0}^{\infty} \gamma^{k}r_{t+k+1} \mid s_t = s, a_t = a \right]$$</span></p>
  <p>It is used for the calculation of Q values.</p>
  <p>I can understand the following</p>
  <ol>
  <li><p><span class="math-container">$A$</span> is <span class="math-container">$\{s_t = s, a_t = a\}$</span> .i.e., agent has been performed action <span class="math-container">$a$</span> on state <span class="math-container">$s$</span> at time step <span class="math-container">$t$</span> and</p>
  </li>
  <li><p><span class="math-container">$g(X)$</span> is <span class="math-container">$\sum\limits_{k=0}^{\infty} \gamma^{k}r_{t+k+1}$</span> i.e., return (long run reward).</p>
  </li>
  </ol>
  <p>What I didn't understand is what is <span class="math-container">$X$</span> here. i.e., what is the random variable on which we are calculating long-run rewards?</p>
  <p>My guess is policy function. It is averaging long-run rewards over all possible policy functions. Is it true?</p>
  
Tags: <reinforcement-learning><q-learning><value-functions><notation><random-variable>
Title: In the definition of the state-action value function, what is the random variable we take the expectation of?
UserDisplayName: hanugm
ThreadText: # In the definition of the state-action value function, what is the random variable we take the expectation of?
  
  (Asked by: username_0 on 2020-10-17)
  
  I know that
  
  $$\mathbb{E}[g(X) \mid A] = \sum\limits\_{x} g(x) p\_{X \mid A}(x)$$
  
  for any random variable $X$.
  
  Now, consider the following expression.
  
  $$\mathbb{E}\_{\pi} \left[ \sum \limits\_{k=0}^{\infty} \gamma^{k}r\_{t+k+1} \mid s\_t = s, a\_t = a \right]$$
  
  It is used for the calculation of Q values.
  
  I can understand the following
  
  1. $A$ is $\{s\_t = s, a\_t = a\}$ .i.e., agent has been performed action $a$ on state $s$ at time step $t$ and
  2. $g(X)$ is $\sum\limits\_{k=0}^{\infty} \gamma^{k}r\_{t+k+1}$ i.e., return (long run reward).
  
  What I didn't understand is what is $X$ here. i.e., what is the random variable on which we are calculating long-run rewards?
  
  My guess is policy function. It is averaging long-run rewards over all possible policy functions. Is it true?
  
  **Tags:** reinforcement-learning, q-learning, value-functions, notation, random-variable
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-17. Score: 2 | Accepted answer
  
  I am using the convention of uppercase $X$ for random variable and lowercase $x$ for an individual observation. It is possible your source material did not do this, which might be causing your confusion. However, it is the convention used in Sutton & Barto's [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2020.pdf).
  
  > What I didn't understand is what is here. i.e., what is the random variable on which we are calculating long-run rewards?
  
  The random variable is $R\_t$, the reward at each time step. The distribution of $R\_t$ in turn depends on the distribution of $S\_{t-1}$ and $A\_{t-1}$ plus the policy and state progression rules. There is no need to include the *process* that causes the distribution of each $R\_t$ in every equation. Although sometimes it is useful to do so, for example when deriving the Bellman equations for value functions.
  
  > My guess is policy function. It is averaging long-run rewards over all possible policy functions. Is it true?
  
  No, this is not true. In fact, it is the more usual assumption that the policy function $\pi(a|s)$ remains constant over the expectation, and this is what the subscript $\pi$ in $\mathbb{E}\_{\pi}[...]$ means.
  
  The expectation is over randomness due to the policy $\pi$, plus randomness due to the environment, which can be described by the function $p(r, s'|s, a)$ - the probability of observing reward $r$ and next state $s'$ given starting in state $s$ and taking action $a$. These two functions combine to create the distribution of $R\_t$. It is possible that both functions are deterministic in practice, thus $R\_t$ is also deterministic. However, RL theory works on the more general stochastic case, which is also used to model exploratory actions, even if the target policy and environment are deterministic.
  
  > **Comments:**
  > **username_2**: To me, it's not fully clear what you mean by this sentence "_There is no need to confuse the process that causes the distribution of each $R_t$ (which can be broken down analyitically, and this is done to derive the Bellman equations), with the fact that you are measuring $R_t$ directly in the expectation._"  
  > **username_1**: @username_2 I've had an attempt to reword that paragraph.  
  > **username_0**: If $R$ is the random variable, then all the possible values of $R, (r_{t+1}, r_{t+2}.....)$ are used in the summation. Then on which values does the expectation  is calculated on? I tried a lot, but can't able to figure out where I am going wrong. Is it on all possible reward functions?  
  > **username_1**: @username_0 If $R_t$ is a random variable, it has a _distribution_, The expectation is calculated over that distribution. The precise distribution depends on the policy being used, plus the state progression and reward function from the environment. Typically in RL you don't work with the full distribution of the return ($G_t$ or sum of rewards), although that may be possible for very simple environments. Instead you work either with samples from the distribution (e.g. Monte Carlo methods) or the Bellman equation (e.g. Value Iteration) or both (e.g. TD learning).  
  > **username_0**: So, If I have a fixed Immediate reward function and policy function, then there is no need of the expectation. Is it right?  
  > **username_1**: @username_0:  You would need a fixed *deterministic* policy, a *deterministic* reward function and a *deterministic* state progression. Then in theory all sampled results would exactly equal the expectation. That is not a usual setup for RL, because you also want to explore and typically that is done by having a stochastic policy.  
  > **username_1**: @username_0: The equations with the expectation are used to figure out RL theory.  They are directly relevant for methods like Value Iteration. Other than that, unless you are working on the thoery (studying existing approaches, or devising new ones) then you don't need to work with it directly.  
  
  ---
  
  