Answers.Body: <p>Let's say we have <span class="math-container">$a$</span> - constant and <span class="math-container">$\epsilon \sim \mathcal{N}(0,\sigma)$</span>, then:
  <span class="math-container">$$\mathbb{E}\left[(a+\epsilon)^2\right] = \mathbb{E}\left[a^2\right] + 2 \mathbb{E}\left[a\right]\mathbb{E}\left[\epsilon\right] + \mathbb{E}\left[\epsilon^2\right] $$</span>
  Expectations of constants are just the constants: <span class="math-container">$\mathbb{E}[a] = a$</span> and  <span class="math-container">$\mathbb{E}[a^2] = a^2$</span><br />
  The mean of <span class="math-container">$\epsilon$</span> is zero <span class="math-container">$\mathbb{E}[\epsilon] = 0$</span>. And the expectation of <span class="math-container">$\epsilon^2$</span> is its variance:
  <span class="math-container">$$ \mathop{\mathrm{Var}}(\epsilon)  = \mathbb{E}[\epsilon^2] - \mathbb{E}[\epsilon]^2 = \mathbb{E}[\epsilon^2]$$</span></p>
  <p>Substituting, we get an expression for the original expectation:<br />
  <span class="math-container">$$\mathbb{E}\left[(a+\epsilon)^2\right] = a^2 + \mathop{\mathrm{Var}}(\epsilon) \tag{*}$$</span></p>
  <p>Getting to the expectation in the book, we first substitute the values for <span class="math-container">$Y$</span> and <span class="math-container">$\hat{Y}$</span>:
  <span class="math-container">$$\mathbb{E}\left[(Y - \hat{Y})^2\right] = \mathbb{E}\left[((f(X) - \hat{f}(X)) + \epsilon)^2\right]$$</span></p>
  <p>In the book it is assumed that <span class="math-container">$X$</span>, <span class="math-container">$f$</span> and <span class="math-container">$\hat{f}$</span> are constant. So we can use the expression (*) with the constant being <span class="math-container">$a = f(X) - \hat{f}(X)$</span>:</p>
  <p><span class="math-container">$$\mathbb{E}\left[(Y - \hat{Y})^2\right] = (f(X) - \hat{f}(X))^2 + \mathop{\mathrm{Var}}(\epsilon)$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-16T18:59:10.600Z
Answers.LastActivityDate: 2021-05-18T09:30:25.607Z
Answers.LastEditDate: 2021-05-18T09:30:25.607Z
Answers.UserDisplayName: Kostya
Answers.Body: <p>Let me try to show this. The only (non-constant) random variable here is <span class="math-container">$\epsilon$</span>, while <span class="math-container">$f(X)$</span> and <span class="math-container">$\hat{Y} = \hat{f}(X)$</span> are constant random variables (so their expectations is equal to their only value).</p>
  <p>So, we start with the following expression.</p>
  <p><span class="math-container">\begin{align}
  \mathbb{E} \left[ (Y - \hat{Y})^2 \right] \tag{1}\label{1}
  \end{align}</span></p>
  <p>Now, we just apply the distributive property, so \ref{1} becomes</p>
  <p><span class="math-container">\begin{align}
  \mathbb{E} \left[ Y^2 - 2Y \hat{Y} +  \hat{Y}^2 \right] \tag{2}\label{2}
  \end{align}</span></p>
  <p>Given the linearity of the expectation, we can write \ref{2} as follows</p>
  <p><span class="math-container">\begin{align}
  \mathbb{E} \left[ Y^2 \right]  - \mathbb{E} \left[ 2Y \hat{Y} \right]  +  \mathbb{E} \left[\hat{Y}^2 \right] \tag{3}\label{3}
  \end{align}</span></p>
  <p>Given that <span class="math-container">$\hat{Y} = \hat{f}(X)$</span> is a constant and that we can take constants out of the expectations, we have</p>
  <p><span class="math-container">\begin{align}
  \mathbb{E} \left[ Y^2 \right]  - 2 \hat{Y} \mathbb{E} \left[ Y  \right]  +  \hat{Y}^2 \tag{4}\label{4}
  \end{align}</span></p>
  <p>Now, let's replace <span class="math-container">$Y$</span> with <span class="math-container">$f(X) + \epsilon$</span>, to obtain</p>
  <p><span class="math-container">\begin{align}
  \mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right]  - 2 \hat{Y} \mathbb{E} \left[ f(X) + \epsilon \right]  +  \hat{Y}^2 \tag{5}\label{5}
  \end{align}</span></p>
  <p>Now, in the book, they assume that <span class="math-container">$\epsilon \sim \mathcal{N}(0, \sigma)$</span>, so <span class="math-container">$\mathbb{E}\left[ \epsilon \right] = 0$</span> (i.e. the expected value of <span class="math-container">$\epsilon$</span> is just the mean of the Gaussian, which is assumed to be zero). So, \ref{5} becomes</p>
  <p><span class="math-container">\begin{align}
  &amp;\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right]  - 2 \hat{Y} \left( \mathbb{E} \left[ f(X) \right] + \mathbb{E} \left[ \epsilon \right] \right)  +  \hat{Y}^2
  = \\
  &amp;\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right]  - 2 \hat{Y} \left(  f(X) + 0 \right)  +  \hat{Y}^2 
  = \\
  &amp;\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right]  - 2 \hat{Y} f(X)  +  \hat{Y}^2
  = \\
  &amp;\mathbb{E} \left[  f(X)^2 + 2 f(X) \epsilon  + \epsilon^2 \right]  - 2 \hat{Y} f(X)  +  \hat{Y}^2 =
  \\
  &amp;\mathbb{E} \left[  f(X)^2 \right]  + \mathbb{E} \left[  2 f(X) \epsilon \right]   + \mathbb{E} \left[  \epsilon^2 \right]  - 2 \hat{Y} f(X)  +  \hat{Y}^2 = \\
  &amp; \mathbb{E} \left[  \epsilon^2 \right]  + f(X)^2  - 2 \hat{Y} f(X)  +  \hat{Y}^2 
  = 
  \\
  &amp;
  \mathbb{E} \left[  \epsilon^2 \right]  + \left(f(X)  - \hat{Y} \right)^2  \tag{6}\label{6}
  \end{align}</span></p>
  <p>Now, note that the <a href="https://en.wikipedia.org/wiki/Variance" rel="nofollow noreferrer">variance of a random variable <span class="math-container">$Z$</span></a> is defined as</p>
  <p><span class="math-container">$$\operatorname {Var} (Z)=\mathbb {E} \left[(Z - \mu_Z )^{2}\right]$$</span></p>
  <p>In our case, <span class="math-container">$\mu_Z$</span> is zero, so the variance of <span class="math-container">$\epsilon$</span> is <span class="math-container">$\mathbb{E} \left[  \epsilon^2 \right]$</span>, so \ref{6} becomes</p>
  <p><span class="math-container">\begin{align}
  \operatorname{Var} (\epsilon)  + \left(f(X)  - \hat{Y} \right)^2 \\
   \tag{7}\label{7}
  \end{align}</span></p>
  <p>You can also come up with the same result in a different and simpler way, i.e. rewrite <span class="math-container">$\mathbb{E}\left[ \left( f(X) + \epsilon - \hat f(X) \right)^2 \right]$</span> as <span class="math-container">$\mathbb{E}\left[ \left( \left(f(X) - \hat f(X)\right) +\epsilon  \right)^2 \right]$</span>, then you apply the distributive property and similar rules that I applied above to derive the same result.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-18T09:08:27.850Z
Answers.Text: Meanwhile, I also realized that this question was also asked [here](https://stats.stackexchange.com/q/191113). You can also find the proofs of equation 2.7 of the same book [here](https://stats.stackexchange.com/q/204115). There's also [this Wikipedia article](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) that talks about the topic.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-16T16:04:56.600Z
Answers.LastActivityDate: 2021-05-16T19:01:07.907Z
Answers.LastEditDate: 2021-05-16T19:01:07.907Z
Answers.UserDisplayName: nbro
Body: <p>In the book <a href="https://www.statlearning.com/" rel="nofollow noreferrer">An Introduction to Statistical Learning</a>, the authors claim (equation 2.3, p. 19, chapter 2)</p>
  <p><span class="math-container">$$\mathbb{E} \left[ (Y - \hat{Y})^2 \right] =  \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon) \label{0}\tag{0},$$</span></p>
  <p>where</p>
  <ul>
  <li><span class="math-container">$Y = f(X) + \epsilon$</span>, where <span class="math-container">$\epsilon \sim \mathcal{N}(0, \sigma)$</span> and <span class="math-container">$f$</span> is the unknown function we want to estimate</li>
  <li><span class="math-container">$\hat{Y} = \hat{f}(X)$</span> is the output of our estimate of <span class="math-container">$f$</span>, i.e. <span class="math-container">$\hat{f} \approx f$</span></li>
  </ul>
  <p>They claim that this is easy to prove, but this may not be easy to prove for everyone. So, why is equation \ref{0} true?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-16T19:11:30.150Z
Comments.Text: This is missing a crucial assumption in the book that $X$ (and both $f$s) are fixed. Otherwise the equation is not true.
Comments.UserDisplayName: Kostya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-16T19:51:31.850Z
Comments.Text: @Kostya I'm saying that in my answer.
Comments.UserDisplayName: nbro
Tags: <machine-learning><proofs><function-approximation><statistical-ai><probability-theory>
Title: Why is the equation $\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon)$ true?
UserDisplayName: nbro
ThreadText: # Why is the equation $\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon)$ true?
  
  (Asked by: username_0 on 2021-05-16)
  
  In the book [An Introduction to Statistical Learning](https://www.statlearning.com/), the authors claim (equation 2.3, p. 19, chapter 2)
  
  $$\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon) \label{0}\tag{0},$$
  
  where
  
  * $Y = f(X) + \epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma)$ and $f$ is the unknown function we want to estimate
  * $\hat{Y} = \hat{f}(X)$ is the output of our estimate of $f$, i.e. $\hat{f} \approx f$
  
  They claim that this is easy to prove, but this may not be easy to prove for everyone. So, why is equation \ref{0} true?
  
  **Tags:** machine-learning, proofs, function-approximation, statistical-ai, probability-theory
  
  > **Comments:**
  > **username_1**: This is missing a crucial assumption in the book that $X$ (and both $f$s) are fixed. Otherwise the equation is not true.  
  > **username_0**: @username_1 I'm saying that in my answer.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-05-16. Score: 2 | Accepted answer
  
  Let's say we have $a$ - constant and $\epsilon \sim \mathcal{N}(0,\sigma)$, then:
  $$\mathbb{E}\left[(a+\epsilon)^2\right] = \mathbb{E}\left[a^2\right] + 2 \mathbb{E}\left[a\right]\mathbb{E}\left[\epsilon\right] + \mathbb{E}\left[\epsilon^2\right] $$
  Expectations of constants are just the constants: $\mathbb{E}[a] = a$ and $\mathbb{E}[a^2] = a^2$  
  The mean of $\epsilon$ is zero $\mathbb{E}[\epsilon] = 0$. And the expectation of $\epsilon^2$ is its variance:
  $$ \mathop{\mathrm{Var}}(\epsilon) = \mathbb{E}[\epsilon^2] - \mathbb{E}[\epsilon]^2 = \mathbb{E}[\epsilon^2]$$
  
  Substituting, we get an expression for the original expectation:  
  $$\mathbb{E}\left[(a+\epsilon)^2\right] = a^2 + \mathop{\mathrm{Var}}(\epsilon) \tag{\*}$$
  
  Getting to the expectation in the book, we first substitute the values for $Y$ and $\hat{Y}$:
  $$\mathbb{E}\left[(Y - \hat{Y})^2\right] = \mathbb{E}\left[((f(X) - \hat{f}(X)) + \epsilon)^2\right]$$
  
  In the book it is assumed that $X$, $f$ and $\hat{f}$ are constant. So we can use the expression (\*) with the constant being $a = f(X) - \hat{f}(X)$:
  
  $$\mathbb{E}\left[(Y - \hat{Y})^2\right] = (f(X) - \hat{f}(X))^2 + \mathop{\mathrm{Var}}(\epsilon)$$
  
  
  ---
  
  ### Answer by username_0 on 2021-05-16. Score: 1
  
  Let me try to show this. The only (non-constant) random variable here is $\epsilon$, while $f(X)$ and $\hat{Y} = \hat{f}(X)$ are constant random variables (so their expectations is equal to their only value).
  
  So, we start with the following expression.
  
  \begin{align}
  \mathbb{E} \left[ (Y - \hat{Y})^2 \right] \tag{1}\label{1}
  \end{align}
  
  Now, we just apply the distributive property, so \ref{1} becomes
  
  \begin{align}
  \mathbb{E} \left[ Y^2 - 2Y \hat{Y} + \hat{Y}^2 \right] \tag{2}\label{2}
  \end{align}
  
  Given the linearity of the expectation, we can write \ref{2} as follows
  
  \begin{align}
  \mathbb{E} \left[ Y^2 \right] - \mathbb{E} \left[ 2Y \hat{Y} \right] + \mathbb{E} \left[\hat{Y}^2 \right] \tag{3}\label{3}
  \end{align}
  
  Given that $\hat{Y} = \hat{f}(X)$ is a constant and that we can take constants out of the expectations, we have
  
  \begin{align}
  \mathbb{E} \left[ Y^2 \right] - 2 \hat{Y} \mathbb{E} \left[ Y \right] + \hat{Y}^2 \tag{4}\label{4}
  \end{align}
  
  Now, let's replace $Y$ with $f(X) + \epsilon$, to obtain
  
  \begin{align}
  \mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} \mathbb{E} \left[ f(X) + \epsilon \right] + \hat{Y}^2 \tag{5}\label{5}
  \end{align}
  
  Now, in the book, they assume that $\epsilon \sim \mathcal{N}(0, \sigma)$, so $\mathbb{E}\left[ \epsilon \right] = 0$ (i.e. the expected value of $\epsilon$ is just the mean of the Gaussian, which is assumed to be zero). So, \ref{5} becomes
  
  \begin{align}
  &\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} \left( \mathbb{E} \left[ f(X) \right] + \mathbb{E} \left[ \epsilon \right] \right) + \hat{Y}^2
  = \\
  &\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} \left( f(X) + 0 \right) + \hat{Y}^2
  = \\
  &\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} f(X) + \hat{Y}^2
  = \\
  &\mathbb{E} \left[ f(X)^2 + 2 f(X) \epsilon + \epsilon^2 \right] - 2 \hat{Y} f(X) + \hat{Y}^2 =
  \\
  &\mathbb{E} \left[ f(X)^2 \right] + \mathbb{E} \left[ 2 f(X) \epsilon \right] + \mathbb{E} \left[ \epsilon^2 \right] - 2 \hat{Y} f(X) + \hat{Y}^2 = \\
  & \mathbb{E} \left[ \epsilon^2 \right] + f(X)^2 - 2 \hat{Y} f(X) + \hat{Y}^2
  =
  \\
  &
  \mathbb{E} \left[ \epsilon^2 \right] + \left(f(X) - \hat{Y} \right)^2 \tag{6}\label{6}
  \end{align}
  
  Now, note that the [variance of a random variable $Z$](https://en.wikipedia.org/wiki/Variance) is defined as
  
  $$\operatorname {Var} (Z)=\mathbb {E} \left[(Z - \mu\_Z )^{2}\right]$$
  
  In our case, $\mu\_Z$ is zero, so the variance of $\epsilon$ is $\mathbb{E} \left[ \epsilon^2 \right]$, so \ref{6} becomes
  
  \begin{align}
  \operatorname{Var} (\epsilon) + \left(f(X) - \hat{Y} \right)^2 \\
  \tag{7}\label{7}
  \end{align}
  
  You can also come up with the same result in a different and simpler way, i.e. rewrite $\mathbb{E}\left[ \left( f(X) + \epsilon - \hat f(X) \right)^2 \right]$ as $\mathbb{E}\left[ \left( \left(f(X) - \hat f(X)\right) +\epsilon \right)^2 \right]$, then you apply the distributive property and similar rules that I applied above to derive the same result.
  
  > **Comments:**
  > **username_0**: Meanwhile, I also realized that this question was also asked [here](https://stats.stackexchange.com/q/191113). You can also find the proofs of equation 2.7 of the same book [here](https://stats.stackexchange.com/q/204115). There's also [this Wikipedia article](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) that talks about the topic.  
  
  ---
  
  