Answers.Body: <h2>Mathematical Interpretation</h2>
  <p>Note that equation (2.23) is simply calculating the conditional distribution of equation (2.21) and then finding the mean. Your question reduces to:</p>
  <p>&quot;Given normal variables <span class="math-container">$X$</span> and <span class="math-container">$Y$</span>, why is <span class="math-container">$\mathbb{E}[Y|X] = \mu_y + Cov(X, Y)Cov(Y, Y)^{-1}(x - \mu_x)$</span>? (note: in the book, <span class="math-container">$\mu_x = \mu_y = 0$</span>)</p>
  <p>Deriving the conditional probability mean is complicated (see <a href="http://home.iitk.ac.in/%7Ezeeshan/pdf/The%20Bivariate%20Normal%20Distribution.pdf" rel="nofollow noreferrer">The Bivariate Normal Distribution</a>, page 3). A more intuitive look can be seen in the first graph in <a href="https://online.stat.psu.edu/stat414/lesson/21/21.1" rel="nofollow noreferrer">this page</a>.</p>
  <p>Here, the mean of <span class="math-container">$Y|X$</span> is linear in what value <span class="math-container">$X$</span> takes. The line starts at the intercept <span class="math-container">$\mu_y$</span>, and increases with slope <span class="math-container">$\rho (\frac{\sigma_y}{\sigma_x}) = \frac{Cov(X,Y)}{\sigma_x \sigma_y}\left(\frac{\sigma_y}{\sigma_x}\right)=\frac{Cov(X,Y)}{\sigma_x^2} = \frac{Cov(X,Y)}{Var(x)} = \frac{Cov(X,Y)}{Cov(X,X)}$</span>. So the mathematical interpretation of <span class="math-container">$Cov(X,Y)Cov(X,X)^{-1}$</span> is that <strong>it is the slope of the relationship between the mean of <span class="math-container">$Y|X$</span> and the value of <span class="math-container">$x$</span> that you are given</strong>. As you are given a higher value <span class="math-container">$x$</span>, say <span class="math-container">$x + \delta$</span>, then the mean of <span class="math-container">$Y|X$</span> raises by <span class="math-container">$Cov(X,Y)Cov(X,X)^{-1}\delta$</span>.</p>
  <p>Why is there even a <span class="math-container">$Cov(X,Y)Cov(X,X)^{-1}$</span> term there? For some reason, multiplying <span class="math-container">$Y|X$</span> by a <span class="math-container">$Cov(X,Y)Cov(X,X)^{-1}$</span> term makes <span class="math-container">$Y|X$</span> completely independent of <span class="math-container">$X$</span> (which makes sense as the definition of &quot;conditional probability&quot;, because you are already given a value of <span class="math-container">$X$</span>). This is just a mathematical property, I don't know if there's an intuitive explanation as to why.</p>
  <h2>Human Interpretation</h2>
  <p>In case your post just want an intuition as to why there is a <span class="math-container">$Cov(X,X)^{-1}$</span> in the prediction of a Gaussian process (and ignoring the conditional probability fluff), I don't think there's a real basis for this, it would only be coincidental as the authors simply used the conditional probability mean formula, but I would guess <strong><span class="math-container">$Cov(X,X)^{-1}$</span> somehow normalizes the values of covariance matrix <span class="math-container">$Cov(X_*, X)$</span></strong>.</p>
  <p>For example, if the training set <span class="math-container">$X$</span> has a lot of outliers and therefore extremely high variance (e.g. all non-diagonal entries in millions), then it is very likely that <span class="math-container">$Cov(X_*, X)$</span> would also be extremely high as <span class="math-container">$X_*$</span> follows the same distribution as <span class="math-container">$X$</span> (unless each data in <span class="math-container">$X_*$</span> matches the exact same variance in <span class="math-container">$X$</span>). It doesn't make sense to multiply <span class="math-container">$y$</span> by millions though, as <span class="math-container">$y$</span> is already a somewhat decent estimator/prior.</p>
  <p>It makes more sense to normalize <span class="math-container">$Cov(X_*, X)$</span> by dividing it with the training data variance <span class="math-container">$Cov(X,X) = Var(X)$</span> so that the ratio <span class="math-container">$Cov(X_*, X)Cov(X, X)^{-1}$</span> tends to be more closer to 1 when <span class="math-container">$X_*$</span> follows the same distribution as <span class="math-container">$X$</span> (which should be the case). If the ratio is exactly 1, then <span class="math-container">$X_*$</span> has the exact same distribution as <span class="math-container">$X$</span>, so you just return the prior estimate <span class="math-container">$y$</span>. If the ratio is far away from <span class="math-container">$1$</span>, then the test set distribution is wildly different than the training set distribution, so you return a number far away from <span class="math-container">$y$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-14T20:21:10.940Z
Answers.LastActivityDate: 2020-12-14T20:21:10.940Z
Answers.UserDisplayName: user3667125
Answers.Body: <p>I interpret this as the following, I will use the uppercase matrix notation <span class="math-container">$\mathbf{K_{*}}$</span>, etc...</p>
  <p>The covariance matrix <span class="math-container">$\mathbf{K_{xx}}$</span> summarizes everything we know about the input feature space. I think of it as a unique signature that describes the data we have in <span class="math-container">$\mathbb{R}^{d_x}$</span>. Along with the example training data, we have the labels <span class="math-container">$\mathbf{y}$</span>, which give us a concrete definition of what the prediction values for <span class="math-container">$\mathbf{K_{xx}}$</span> should be.</p>
  <p>We can then ask the question &quot;What can we multiply the unique signature <span class="math-container">$\mathbf{K_{xx}}$</span> by in order to get the training outputs?&quot; This takes the familiar form of <span class="math-container">$\mathbf{A}\mathbf{x} = \mathbf{b}$</span>...</p>
  <p><span class="math-container">$$
  \begin{aligned}
  \mathbf{K}_{xx}\mathbf{z} &amp;= \mathbf{y} \\
  \mathbf{z} &amp;= \mathbf{K}_{xx}^{-1}\mathbf{y}
  \end{aligned}
  $$</span></p>
  <p><span class="math-container">$\mathbf{z}$</span> then represents the vector that the covariance matrix transforms into the outputs. Then all we have to do is multiply the new covariance of the train/test points by that same vector to get the predictions...</p>
  <p><span class="math-container">$$
  \mathbf{\hat{y}} = \mathbf{K_{*x}}\mathbf{K_{xx}^{-1}}\mathbf{y}
  $$</span></p>
  <p>I should add that I am relatively certain about this, but I am still learning this stuff myself so I am not 100</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-14T12:37:10.777Z
Answers.LastActivityDate: 2020-11-14T12:37:10.777Z
Answers.UserDisplayName: Joff
Body: <p>The formula for mean prediction using Gaussian Process is <span class="math-container">$k(x_*, x)k(x, x)^{-1}y$</span>, where <span class="math-container">$k$</span> is the covariance function. See e.g. equation 2.23 (in chapter 2) from <a href="http://www.gaussianprocess.org/gpml/chapters/RW2.pdf" rel="noreferrer">Gaussian Processes for Machine Learning</a> (2006) by C. E. Rasmussen &amp; C. K. I. Williams.</p>
  
  <p>Oversimplifying, the mean prediction of the new point <span class="math-container">$y_*$</span> is the weighted average of previously observed <span class="math-container">$y$</span>, where the weights are calculated by the <span class="math-container">$k(x_*,x)$</span> and normalized by <span class="math-container">$k(x,x)^{-1}$</span>. </p>
  
  <p>Now, the first part <span class="math-container">$k(x_*, x)$</span> is easy to interpret. The closer the new data point lies to the previously observed data points, the greater their similarity, the higher will be the weight and impact on the prediction.</p>
  
  <p>But how to interpret the second part <span class="math-container">$k(x, x)^{-1}$</span>? I presume this makes the weight of the points in the clusters greater than the outliers. Am I correct?</p>
  
Tags: <math><gaussian-process>
Title: Interpretation of inverse matrix in mean calculation in Gaussian Process
UserDisplayName: Kasia
ThreadText: # Interpretation of inverse matrix in mean calculation in Gaussian Process
  
  (Asked by: username_0 on 2020-03-14)
  
  The formula for mean prediction using Gaussian Process is $k(x\_\*, x)k(x, x)^{-1}y$, where $k$ is the covariance function. See e.g. equation 2.23 (in chapter 2) from [Gaussian Processes for Machine Learning](http://www.gaussianprocess.org/gpml/chapters/RW2.pdf) (2006) by C. E. Rasmussen & C. K. I. Williams.
  
  Oversimplifying, the mean prediction of the new point $y\_\*$ is the weighted average of previously observed $y$, where the weights are calculated by the $k(x\_\*,x)$ and normalized by $k(x,x)^{-1}$.
  
  Now, the first part $k(x\_\*, x)$ is easy to interpret. The closer the new data point lies to the previously observed data points, the greater their similarity, the higher will be the weight and impact on the prediction.
  
  But how to interpret the second part $k(x, x)^{-1}$? I presume this makes the weight of the points in the clusters greater than the outliers. Am I correct?
  
  **Tags:** math, gaussian-process
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-12-14. Score: 1
  
  ## Mathematical Interpretation
  
  Note that equation (2.23) is simply calculating the conditional distribution of equation (2.21) and then finding the mean. Your question reduces to:
  
  "Given normal variables $X$ and $Y$, why is $\mathbb{E}[Y|X] = \mu\_y + Cov(X, Y)Cov(Y, Y)^{-1}(x - \mu\_x)$? (note: in the book, $\mu\_x = \mu\_y = 0$)
  
  Deriving the conditional probability mean is complicated (see [The Bivariate Normal Distribution](http://home.iitk.ac.in/%7Ezeeshan/pdf/The%20Bivariate%20Normal%20Distribution.pdf), page 3). A more intuitive look can be seen in the first graph in [this page](https://online.stat.psu.edu/stat414/lesson/21/21.1).
  
  Here, the mean of $Y|X$ is linear in what value $X$ takes. The line starts at the intercept $\mu\_y$, and increases with slope $\rho (\frac{\sigma\_y}{\sigma\_x}) = \frac{Cov(X,Y)}{\sigma\_x \sigma\_y}\left(\frac{\sigma\_y}{\sigma\_x}\right)=\frac{Cov(X,Y)}{\sigma\_x^2} = \frac{Cov(X,Y)}{Var(x)} = \frac{Cov(X,Y)}{Cov(X,X)}$. So the mathematical interpretation of $Cov(X,Y)Cov(X,X)^{-1}$ is that **it is the slope of the relationship between the mean of $Y|X$ and the value of $x$ that you are given**. As you are given a higher value $x$, say $x + \delta$, then the mean of $Y|X$ raises by $Cov(X,Y)Cov(X,X)^{-1}\delta$.
  
  Why is there even a $Cov(X,Y)Cov(X,X)^{-1}$ term there? For some reason, multiplying $Y|X$ by a $Cov(X,Y)Cov(X,X)^{-1}$ term makes $Y|X$ completely independent of $X$ (which makes sense as the definition of "conditional probability", because you are already given a value of $X$). This is just a mathematical property, I don't know if there's an intuitive explanation as to why.
  
  ## Human Interpretation
  
  In case your post just want an intuition as to why there is a $Cov(X,X)^{-1}$ in the prediction of a Gaussian process (and ignoring the conditional probability fluff), I don't think there's a real basis for this, it would only be coincidental as the authors simply used the conditional probability mean formula, but I would guess **$Cov(X,X)^{-1}$ somehow normalizes the values of covariance matrix $Cov(X\_\*, X)$**.
  
  For example, if the training set $X$ has a lot of outliers and therefore extremely high variance (e.g. all non-diagonal entries in millions), then it is very likely that $Cov(X\_\*, X)$ would also be extremely high as $X\_\*$ follows the same distribution as $X$ (unless each data in $X\_\*$ matches the exact same variance in $X$). It doesn't make sense to multiply $y$ by millions though, as $y$ is already a somewhat decent estimator/prior.
  
  It makes more sense to normalize $Cov(X\_\*, X)$ by dividing it with the training data variance $Cov(X,X) = Var(X)$ so that the ratio $Cov(X\_\*, X)Cov(X, X)^{-1}$ tends to be more closer to 1 when $X\_\*$ follows the same distribution as $X$ (which should be the case). If the ratio is exactly 1, then $X\_\*$ has the exact same distribution as $X$, so you just return the prior estimate $y$. If the ratio is far away from $1$, then the test set distribution is wildly different than the training set distribution, so you return a number far away from $y$.
  
  
  ---
  
  ### Answer by username_2 on 2020-11-14. Score: 0
  
  I interpret this as the following, I will use the uppercase matrix notation $\mathbf{K\_{\*}}$, etc...
  
  The covariance matrix $\mathbf{K\_{xx}}$ summarizes everything we know about the input feature space. I think of it as a unique signature that describes the data we have in $\mathbb{R}^{d\_x}$. Along with the example training data, we have the labels $\mathbf{y}$, which give us a concrete definition of what the prediction values for $\mathbf{K\_{xx}}$ should be.
  
  We can then ask the question "What can we multiply the unique signature $\mathbf{K\_{xx}}$ by in order to get the training outputs?" This takes the familiar form of $\mathbf{A}\mathbf{x} = \mathbf{b}$...
  
  $$
  \begin{aligned}
  \mathbf{K}\_{xx}\mathbf{z} &= \mathbf{y} \\
  \mathbf{z} &= \mathbf{K}\_{xx}^{-1}\mathbf{y}
  \end{aligned}
  $$
  
  $\mathbf{z}$ then represents the vector that the covariance matrix transforms into the outputs. Then all we have to do is multiply the new covariance of the train/test points by that same vector to get the predictions...
  
  $$
  \mathbf{\hat{y}} = \mathbf{K\_{\*x}}\mathbf{K\_{xx}^{-1}}\mathbf{y}
  $$
  
  I should add that I am relatively certain about this, but I am still learning this stuff myself so I am not 100
  
  
  ---
  
  