Answers.Body: <p><span class="math-container">$\mathbb E$</span> is the symbol for the <a href="https://en.wikipedia.org/wiki/Expected_value" rel="nofollow noreferrer">expectation (or expected value)</a>. </p>
  
  <p>To fully understand the concept of <em>expected value</em>, you need to understand the concept of <a href="https://en.wikipedia.org/wiki/Random_variable" rel="nofollow noreferrer"><em>random variable</em></a>. An example should help you understand the idea behind the concept of a random variable. </p>
  
  <p>Suppose you <a href="https://en.wikipedia.org/wiki/Coin_flipping" rel="nofollow noreferrer">toss a coin</a>. The outcome of this (random) experiment can either be <em>heads</em> or <em>tails</em>. Formally, the <a href="https://en.wikipedia.org/wiki/Sample_space" rel="nofollow noreferrer"><em>sample space</em></a>, <span class="math-container">$\Omega = \{\text{heads}, \text{tails}\}$</span>, is the set that contains the possible outcomes of a random experiment. The outcome (e.g. <em>heads</em>) is the result of a <a href="https://en.wikipedia.org/wiki/Stochastic_process" rel="nofollow noreferrer">random process</a>. A random variable is a function that we can associate with a random process so that we can more formally describe the random process. In this case, we can associate a random variable, <span class="math-container">$T$</span>, with this random process of tossing a coin.</p>
  
  <p><span class="math-container">$$
  T(\omega) =
  \begin{cases}
  1, &amp; \text{if } \omega = \text{heads}, \\[6pt]
  0, &amp; \text{if } \omega = \text{tails},
  \end{cases}
  $$</span></p>
  
  <p>where <span class="math-container">$\omega \in \Omega$</span>.</p>
  
  <p>In other words, if the outcome of the random process is <em>heads</em>, then the output of the associated random variable <span class="math-container">$T$</span> is <span class="math-container">$1$</span>, else it is <span class="math-container">$0$</span>.</p>
  
  <p>We can also associate with each random process (and thus with the corresponding random variable) a probability distribution, which, intuitively, describes the probability of occurrence of each possible outcome of the random process. In the case of the coin-flipping random variable (or process), assuming that the coin is "fair", then the following function describes the probability of each outcome of the coin</p>
  
  <p><span class="math-container">$$
  f_T(t) =
  \begin{cases}
  \tfrac 12,&amp; \text{if }t=1,\\[6pt]
  \tfrac 12,&amp; \text{if }t=0,
  \end{cases}
  $$</span></p>
  
  <p>In other words, there is <span class="math-container">$\tfrac 12$</span> probability that the outcome of the random process is <span class="math-container">$1$</span> (heads) and <span class="math-container">$\tfrac 12$</span> probability that it is <span class="math-container">$0$</span> (tails).</p>
  
  <p>If you throw a coin <span class="math-container">$n$</span> times in the air, how many times will it land heads and tails? Of course, it will depend on the experiment. In the first experiment, you might get <span class="math-container">$\frac{3n}{4}$</span> heads and <span class="math-container">$\frac{n}{4}$</span> tails. In the second experiment, you might get <span class="math-container">$\frac{n}{2}$</span> heads and <span class="math-container">$\frac{n}{2}$</span> tails, and so on. If you repeat this experiment an infinite amount of times (of course, we can't do that, but imagine if we could do that), how many times do you <strong>expect</strong> (on average) to get heads and tails? The expected value is the answer to this question.</p>
  
  <p>In the case of the coin-tossing experiment, the outcomes are discrete (heads or tails), consequently, <span class="math-container">$T$</span> is a <em>discrete random variable</em>. In the case of a discrete random variable, the expected value <strong>is defined as</strong> follows</p>
  
  <p><span class="math-container">$$\mathbb E[T] = \sum_{t \in T} p(t) t$$</span></p>
  
  <p>where <span class="math-container">$t$</span> is the outcome of the random variable <span class="math-container">$t$</span> and <span class="math-container">$p(t)$</span> is the probability of such outcome. In other words, the expected value of a random variable <span class="math-container">$T$</span> is defined as a weighted sum of the values it can take, where the weights are the corresponding probabilities of occurrence. So, in the case of the coin-tossing experiment, the expected value is</p>
  
  <p><span class="math-container">\begin{align}
  \mathbb E[T] 
  &amp;= \sum_{t \in T} p(t) t\\
  &amp;= \frac{1}{2}1 + \frac{1}{2} 0\\
  &amp;=\frac{1}{2}
  \end{align}</span></p>
  
  <p>What does <span class="math-container">$\mathbb E[T] = \frac{1}{2}$</span> mean? Intuitively, it means that half of the times the random process produces heads and half of the times it produces tails, assuming it is governed by the probability distribution <span class="math-container">$f_T(t)$</span>.</p>
  
  <p>Note that, if the probability distribution <span class="math-container">$f_T(t)$</span> had been defined differently, then the expected value would also have been different, given that the expected value is defined as a function of the probability of occurrence of each outcome of the random process.</p>
  
  <p>In your specific examples, <span class="math-container">$\mathbb E$</span> is still the symbol for the expected value. For example, in the case of <span class="math-container">$Q^\pi \left(s,a \right) = \mathbb E \left[R_t|s_t = s, a_t = a, \pi \right]$</span>, <span class="math-container">$Q^\pi \left(s,a \right)$</span> is thus defined as the expected value of the random variable <span class="math-container">$R_t$</span>, given that <span class="math-container">$s_t = s$</span>, <span class="math-container">$a_t = a$</span> and the policy is <span class="math-container">$\pi$</span> (so this is actually a <a href="https://en.wikipedia.org/wiki/Conditional_expectation" rel="nofollow noreferrer">conditional expectation</a>). In this specific case, <span class="math-container">$R_t$</span> represents the <em>return</em> at time step <span class="math-container">$t$</span>, which, in reinforcement learning, is defined as</p>
  
  <p><span class="math-container">$$
  R_t = \sum_{k=0}^\infty \gamma^k r_{t+k+1}
  $$</span></p>
  
  <p>where <span class="math-container">$r_{t+k+1} \in \mathbb{R}$</span> is the reward at time step <span class="math-container">$t+k+1$</span>. <span class="math-container">$R_t$</span> a random variable because it is assumed that the underlying environment is a random process.</p>
  
  <p>It is not always easy to intuitively understand the expected value of a random variable. For example, in the case of a coin-flipping random process, the expected value <span class="math-container">$\frac{1}{2}$</span> should be intuitive (given that it is the average of <span class="math-container">$1$</span> and <span class="math-container">$0$</span>), but, in the case of <span class="math-container">$Q^\pi \left(s,a \right)$</span>, at first glance, it is not clear what the expected value should be (hence the need for algorithms such as Q-learning), given that it depends on the rewards, which depend on the dynamics of the environment. However, the intuition behind the concept of the expected value and the calculation (provided the associated random variable is discrete) does not change.</p>
  
  <p>In the case there is more than one random variable involved in the calculation of the expected value, then we also need to specify the random variable the expected value is being calculated with respect to, hence the subscripts of the expected value in your examples. See, for example, <a href="https://stats.stackexchange.com/q/72613/82135">Subscript notation in expectations</a> for more info.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-27T15:41:03.960Z
Answers.LastActivityDate: 2019-11-07T22:03:52.043Z
Answers.LastEditDate: 2019-11-07T22:03:52.043Z
Answers.UserDisplayName: nbro
Answers.Body: <p>That's the <a href="https://en.wikipedia.org/wiki/Expected_value" rel="noreferrer">Expected Value operator</a>. Intuitively, it gives you the value that you would "expect" ("on average") the expression after it (often in square or other brackets) to have. Typically that expression involves some random variables, which means that there may be a wide range of different values the expression may take in any concrete, single event. Taking the expectation basically means that you "average" over all the values the expression could potentially take, appropriately weighted by the probabilities with which certain events occur.</p>
  
  <p>You'll often find assumptions under which the expectation is taken after a vertical line (<span class="math-container">$\mid$</span>) inside the brackets, and/or in the subscript to the right of the <span class="math-container">$\mathbb{E}$</span> symbol. Sometimes, some assumptions may also be left implicit.</p>
  
  <hr>
  
  <p>For example:</p>
  
  <p><span class="math-container">$$\mathbb{E} \left[ R_t \mid s_t = s, a_t = a, \pi \right]$$</span></p>
  
  <p>may be read in english as "the expected value of the discounted returns from time <span class="math-container">$t$</span> onwards (<span class="math-container">$R_t$</span>), given that the state at time <span class="math-container">$t$</span> is <span class="math-container">$s$</span> (<span class="math-container">$s_t = s$</span>), given that our action at time <span class="math-container">$t$</span> is <span class="math-container">$a$</span> (<span class="math-container">$a_t = a$</span>), given that we continue behaving according to policy <span class="math-container">$\pi$</span> after time <span class="math-container">$t$</span>".</p>
  
  <p>I would say that, in this case, the expected value also relies on the transition dynamics of the Markov decision process (i.e. the probabilities with which we transition between states, given our actions). This is left implicit.</p>
  
  <hr>
  
  <p>Second example:</p>
  
  <p><span class="math-container">$$V^{\pi}(s) = \mathbb{E}_{a \sim \pi(s)} \left[ Q^{\pi}(s, a) \right]$$</span></p>
  
  <p>may be read as "<span class="math-container">$V^{\pi}(s)$</span> is equal to the expected value of <span class="math-container">$Q^{\pi}(s, a)$</span>, under the assumption that <span class="math-container">$a$</span> is sampled according to <span class="math-container">$\pi(s)$</span>". </p>
  
  <p>In theory, something like this could be computed by enumerating over all possible <span class="math-container">$a$</span>, computing <span class="math-container">$Q^{\pi}(s, a)$</span> for every such <span class="math-container">$a$</span>, and multiplying it by the probability that <span class="math-container">$\pi(s)$</span> assigns to <span class="math-container">$a$</span>. In practice, you could also approximate it by running a large number of experiments in which you sample <span class="math-container">$a$</span> from <span class="math-container">$\pi(s)$</span>, and then evaluate a single concrete case of <span class="math-container">$Q^{\pi}(s, a)$</span>, and average over all the evaluations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-07T22:17:11.543Z
Answers.Text: Great breakdown "in plain language"--very helpful for those learning the maths!
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-27T15:06:12.793Z
Answers.LastActivityDate: 2019-08-27T15:23:00.037Z
Answers.LastEditDate: 2019-08-27T15:23:00.037Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I came across some papers that use <span class="math-container">$\mathbb E$</span> in equations, in particular, this paper: <a href="https://arxiv.org/pdf/1511.06581.pdf" rel="noreferrer">https://arxiv.org/pdf/1511.06581.pdf</a>. Here is some equations from the paper that uses it: </p>
  
  <p><span class="math-container">$Q^\pi \left(s,a \right) = \mathbb E \left[R_t|s_t = s, a_t = a, \pi \right]$</span> ,</p>
  
  <p><span class="math-container">$V^\pi \left(s \right) = \mathbb E_{a\backsim\pi\left(s \right)} \left[Q^\pi \left(s, a\right) \right]$</span> ,</p>
  
  <p><span class="math-container">$Q^\pi \left(s, a \right) = \mathbb E_{s'} \left[r + \gamma\mathbb E_{a'\backsim\pi \left(s' \right)} \left[Q^\pi \left(s', a' \right) \right] | s,a,\pi \right]$</span></p>
  
  <p><span class="math-container">$\nabla_{\theta_i}L_i\left(\theta_i \right) = \mathbb E_{s, a, r, s'} \left[\left(y_i^{DQN} - Q \left(s, a; \theta_i \right) \right) \nabla_{\theta_i} Q\left(s, a, \theta_i \right) \right]$</span></p>
  
  <p>Could someone explain to me what is the purpose of <span class="math-container">$\mathbb E$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T15:02:18.443Z
Comments.Text: It's the [expected value](https://en.wikipedia.org/wiki/Expected_value) symbol
Comments.UserDisplayName: Brale
Tags: <machine-learning><deep-learning><reinforcement-learning><q-learning>
Title: What does the symbol $\mathbb E$ mean in these equations?
UserDisplayName: theonekeyg
ThreadText: # What does the symbol $\mathbb E$ mean in these equations?
  
  (Asked by: username_0 on 2019-08-27)
  
  I came across some papers that use $\mathbb E$ in equations, in particular, this paper: <https://arxiv.org/pdf/1511.06581.pdf>. Here is some equations from the paper that uses it:
  
  $Q^\pi \left(s,a \right) = \mathbb E \left[R\_t|s\_t = s, a\_t = a, \pi \right]$ ,
  
  $V^\pi \left(s \right) = \mathbb E\_{a\backsim\pi\left(s \right)} \left[Q^\pi \left(s, a\right) \right]$ ,
  
  $Q^\pi \left(s, a \right) = \mathbb E\_{s'} \left[r + \gamma\mathbb E\_{a'\backsim\pi \left(s' \right)} \left[Q^\pi \left(s', a' \right) \right] | s,a,\pi \right]$
  
  $\nabla\_{\theta\_i}L\_i\left(\theta\_i \right) = \mathbb E\_{s, a, r, s'} \left[\left(y\_i^{DQN} - Q \left(s, a; \theta\_i \right) \right) \nabla\_{\theta\_i} Q\left(s, a, \theta\_i \right) \right]$
  
  Could someone explain to me what is the purpose of $\mathbb E$?
  
  **Tags:** machine-learning, deep-learning, reinforcement-learning, q-learning
  
  > **Comments:**
  > **username_1**: It's the [expected value](https://en.wikipedia.org/wiki/Expected_value) symbol  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-08-27. Score: 14 | Accepted answer
  
  That's the [Expected Value operator](https://en.wikipedia.org/wiki/Expected_value). Intuitively, it gives you the value that you would "expect" ("on average") the expression after it (often in square or other brackets) to have. Typically that expression involves some random variables, which means that there may be a wide range of different values the expression may take in any concrete, single event. Taking the expectation basically means that you "average" over all the values the expression could potentially take, appropriately weighted by the probabilities with which certain events occur.
  
  You'll often find assumptions under which the expectation is taken after a vertical line ($\mid$) inside the brackets, and/or in the subscript to the right of the $\mathbb{E}$ symbol. Sometimes, some assumptions may also be left implicit.
  
  ---
  
  For example:
  
  $$\mathbb{E} \left[ R\_t \mid s\_t = s, a\_t = a, \pi \right]$$
  
  may be read in english as "the expected value of the discounted returns from time $t$ onwards ($R\_t$), given that the state at time $t$ is $s$ ($s\_t = s$), given that our action at time $t$ is $a$ ($a\_t = a$), given that we continue behaving according to policy $\pi$ after time $t$".
  
  I would say that, in this case, the expected value also relies on the transition dynamics of the Markov decision process (i.e. the probabilities with which we transition between states, given our actions). This is left implicit.
  
  ---
  
  Second example:
  
  $$V^{\pi}(s) = \mathbb{E}\_{a \sim \pi(s)} \left[ Q^{\pi}(s, a) \right]$$
  
  may be read as "$V^{\pi}(s)$ is equal to the expected value of $Q^{\pi}(s, a)$, under the assumption that $a$ is sampled according to $\pi(s)$".
  
  In theory, something like this could be computed by enumerating over all possible $a$, computing $Q^{\pi}(s, a)$ for every such $a$, and multiplying it by the probability that $\pi(s)$ assigns to $a$. In practice, you could also approximate it by running a large number of experiments in which you sample $a$ from $\pi(s)$, and then evaluate a single concrete case of $Q^{\pi}(s, a)$, and average over all the evaluations.
  
  > **Comments:**
  > **username_4**: Great breakdown "in plain language"--very helpful for those learning the maths!  
  
  ---
  
  ### Answer by username_2 on 2019-08-27. Score: 7
  
  $\mathbb E$ is the symbol for the [expectation (or expected value)](https://en.wikipedia.org/wiki/Expected_value).
  
  To fully understand the concept of *expected value*, you need to understand the concept of [*random variable*](https://en.wikipedia.org/wiki/Random_variable). An example should help you understand the idea behind the concept of a random variable.
  
  Suppose you [toss a coin](https://en.wikipedia.org/wiki/Coin_flipping). The outcome of this (random) experiment can either be *heads* or *tails*. Formally, the [*sample space*](https://en.wikipedia.org/wiki/Sample_space), $\Omega = \{\text{heads}, \text{tails}\}$, is the set that contains the possible outcomes of a random experiment. The outcome (e.g. *heads*) is the result of a [random process](https://en.wikipedia.org/wiki/Stochastic_process). A random variable is a function that we can associate with a random process so that we can more formally describe the random process. In this case, we can associate a random variable, $T$, with this random process of tossing a coin.
  
  $$
  T(\omega) =
  \begin{cases}
  1, & \text{if } \omega = \text{heads}, \\[6pt]
  0, & \text{if } \omega = \text{tails},
  \end{cases}
  $$
  
  where $\omega \in \Omega$.
  
  In other words, if the outcome of the random process is *heads*, then the output of the associated random variable $T$ is $1$, else it is $0$.
  
  We can also associate with each random process (and thus with the corresponding random variable) a probability distribution, which, intuitively, describes the probability of occurrence of each possible outcome of the random process. In the case of the coin-flipping random variable (or process), assuming that the coin is "fair", then the following function describes the probability of each outcome of the coin
  
  $$
  f\_T(t) =
  \begin{cases}
  \tfrac 12,& \text{if }t=1,\\[6pt]
  \tfrac 12,& \text{if }t=0,
  \end{cases}
  $$
  
  In other words, there is $\tfrac 12$ probability that the outcome of the random process is $1$ (heads) and $\tfrac 12$ probability that it is $0$ (tails).
  
  If you throw a coin $n$ times in the air, how many times will it land heads and tails? Of course, it will depend on the experiment. In the first experiment, you might get $\frac{3n}{4}$ heads and $\frac{n}{4}$ tails. In the second experiment, you might get $\frac{n}{2}$ heads and $\frac{n}{2}$ tails, and so on. If you repeat this experiment an infinite amount of times (of course, we can't do that, but imagine if we could do that), how many times do you **expect** (on average) to get heads and tails? The expected value is the answer to this question.
  
  In the case of the coin-tossing experiment, the outcomes are discrete (heads or tails), consequently, $T$ is a *discrete random variable*. In the case of a discrete random variable, the expected value **is defined as** follows
  
  $$\mathbb E[T] = \sum\_{t \in T} p(t) t$$
  
  where $t$ is the outcome of the random variable $t$ and $p(t)$ is the probability of such outcome. In other words, the expected value of a random variable $T$ is defined as a weighted sum of the values it can take, where the weights are the corresponding probabilities of occurrence. So, in the case of the coin-tossing experiment, the expected value is
  
  \begin{align}
  \mathbb E[T]
  &= \sum\_{t \in T} p(t) t\\
  &= \frac{1}{2}1 + \frac{1}{2} 0\\
  &=\frac{1}{2}
  \end{align}
  
  What does $\mathbb E[T] = \frac{1}{2}$ mean? Intuitively, it means that half of the times the random process produces heads and half of the times it produces tails, assuming it is governed by the probability distribution $f\_T(t)$.
  
  Note that, if the probability distribution $f\_T(t)$ had been defined differently, then the expected value would also have been different, given that the expected value is defined as a function of the probability of occurrence of each outcome of the random process.
  
  In your specific examples, $\mathbb E$ is still the symbol for the expected value. For example, in the case of $Q^\pi \left(s,a \right) = \mathbb E \left[R\_t|s\_t = s, a\_t = a, \pi \right]$, $Q^\pi \left(s,a \right)$ is thus defined as the expected value of the random variable $R\_t$, given that $s\_t = s$, $a\_t = a$ and the policy is $\pi$ (so this is actually a [conditional expectation](https://en.wikipedia.org/wiki/Conditional_expectation)). In this specific case, $R\_t$ represents the *return* at time step $t$, which, in reinforcement learning, is defined as
  
  $$
  R\_t = \sum\_{k=0}^\infty \gamma^k r\_{t+k+1}
  $$
  
  where $r\_{t+k+1} \in \mathbb{R}$ is the reward at time step $t+k+1$. $R\_t$ a random variable because it is assumed that the underlying environment is a random process.
  
  It is not always easy to intuitively understand the expected value of a random variable. For example, in the case of a coin-flipping random process, the expected value $\frac{1}{2}$ should be intuitive (given that it is the average of $1$ and $0$), but, in the case of $Q^\pi \left(s,a \right)$, at first glance, it is not clear what the expected value should be (hence the need for algorithms such as Q-learning), given that it depends on the rewards, which depend on the dynamics of the environment. However, the intuition behind the concept of the expected value and the calculation (provided the associated random variable is discrete) does not change.
  
  In the case there is more than one random variable involved in the calculation of the expected value, then we also need to specify the random variable the expected value is being calculated with respect to, hence the subscripts of the expected value in your examples. See, for example, [Subscript notation in expectations](https://stats.stackexchange.com/q/72613/82135) for more info.
  
  
  ---
  
  