Answers.Body: <p>You might want to have a look at the wikipedia article of <a href="https://en.wikipedia.org/wiki/Principal_component_analysis#Further_components" rel="nofollow noreferrer">PCA</a>, where it says:</p>
  <p><em>&quot;The <span class="math-container">$k$</span>th component can be found by subtracting the first <span class="math-container">$k − 1$</span> principal components from <span class="math-container">$\mathbf{X}$</span>:&quot;</em></p>
  <p><span class="math-container">$$\hat{\mathbf{X}}_k = \mathbf{X} - \sum_{s=1}^{k-1}\mathbf{X}\mathbf{w}_s\mathbf{w}_s^T$$</span></p>
  <p>Then you repeat the process to find the next component:</p>
  <p><span class="math-container">$$\mathbf{w}_k = \arg\max \mathbf{w}^T\mathbf{\hat{X}}^T_k\mathbf{\hat{X}}_k\mathbf{w}$$</span>
  <span class="math-container">$$\text{s.t. } \mathbf{w}_k^T\mathbf{w}_k = 1$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-01T09:05:31.870Z
Answers.LastActivityDate: 2020-09-01T09:05:31.870Z
Answers.UserDisplayName: Tinu
Answers.Body: <p>You can also understand the logic from the view of constrained optimisation.
  Introduce a Lagrange function:
  <span class="math-container">$$
  \mathcal{L} = \text{Tr} (w^{T} X X^{T} w) - \lambda w^{T} w
  $$</span>
  And take the derivative with respect to <span class="math-container">$w$</span>:
  <span class="math-container">$$
  \frac{\partial \mathcal{L}}{\partial w} = 2 (X X^{T} - \lambda) w
  $$</span>
  For the general case of dimension <span class="math-container">$\geqslant 1$</span> <span class="math-container">$w$</span> is a set of vectors <span class="math-container">$w = (w_1 w_2 \ldots w_n)$</span>.
  This expression vanishes, if for some index <span class="math-container">$i$</span> <span class="math-container">$w_i$</span> is an of eigenvector of <span class="math-container">$XX^{T}$</span> with the eigenvalue <span class="math-container">$\lambda_i$</span>, and all other components are set to zero. In other words, stationary points are the eigenvectors of <span class="math-container">$X X^{T}$</span>.</p>
  <p>The condititon <span class="math-container">$w^T w = 1$</span> imposes the orthogonality condition on the eigenvectors. In fact, going back to the initial functional, one sees, that <span class="math-container">$w_i X X^{T} w_j = \lambda_j w_i^{T} w_j = 0$</span> for <span class="math-container">$i \neq j$</span>. Therefore, we have finally:
  <span class="math-container">$$
  \mathcal{L} =\sum \lambda_i - \lambda
  $$</span>
  Which is maximized for any <span class="math-container">$k \geq 1$</span>, by taking <span class="math-container">$k$</span> largest eigenvalues.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-01T20:08:00.697Z
Answers.LastActivityDate: 2020-09-01T23:15:52.020Z
Answers.LastEditDate: 2020-09-01T23:15:52.020Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>How does PCA work when we reduce the original space to a 2 or higher-dimensional space? I understand the case when we reduce the dimensionality to <span class="math-container">$1$</span>, but not this case.</p>
  <p><span class="math-container">$$\begin{array}{ll} \text{maximize} &amp; \mathrm{Tr}\left( \mathbf{w}^T\mathbf{X}\mathbf{X}^T\mathbf{w} \right)\\ \text{subject to} &amp; \mathbf{w}^T\mathbf{w} = 1\end{array}$$</span></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-01T22:51:33.927Z
Comments.Text: Related: https://math.stackexchange.com/q/3736092
Comments.UserDisplayName: Rodrigo de Azevedo
Tags: <math><optimization><linear-algebra><principal-component-analysis>
Title: How does PCA work when we reduce the original space to 2 or higher-dimensional space?
UserDisplayName: VN Pikachu
ThreadText: # How does PCA work when we reduce the original space to 2 or higher-dimensional space?
  
  (Asked by: username_0 on 2020-09-01)
  
  How does PCA work when we reduce the original space to a 2 or higher-dimensional space? I understand the case when we reduce the dimensionality to $1$, but not this case.
  
  $$\begin{array}{ll} \text{maximize} & \mathrm{Tr}\left( \mathbf{w}^T\mathbf{X}\mathbf{X}^T\mathbf{w} \right)\\ \text{subject to} & \mathbf{w}^T\mathbf{w} = 1\end{array}$$
  
  **Tags:** math, optimization, linear-algebra, principal-component-analysis
  
  > **Comments:**
  > **username_1**: Related: https://math.stackexchange.com/q/3736092  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-09-01. Score: 2
  
  You might want to have a look at the wikipedia article of [PCA](https://en.wikipedia.org/wiki/Principal_component_analysis#Further_components), where it says:
  
  *"The $k$th component can be found by subtracting the first $k − 1$ principal components from $\mathbf{X}$:"*
  
  $$\hat{\mathbf{X}}\_k = \mathbf{X} - \sum\_{s=1}^{k-1}\mathbf{X}\mathbf{w}\_s\mathbf{w}\_s^T$$
  
  Then you repeat the process to find the next component:
  
  $$\mathbf{w}\_k = \arg\max \mathbf{w}^T\mathbf{\hat{X}}^T\_k\mathbf{\hat{X}}\_k\mathbf{w}$$
  $$\text{s.t. } \mathbf{w}\_k^T\mathbf{w}\_k = 1$$
  
  
  ---
  
  ### Answer by username_3 on 2020-09-01. Score: 0
  
  You can also understand the logic from the view of constrained optimisation.
  Introduce a Lagrange function:
  $$
  \mathcal{L} = \text{Tr} (w^{T} X X^{T} w) - \lambda w^{T} w
  $$
  And take the derivative with respect to $w$:
  $$
  \frac{\partial \mathcal{L}}{\partial w} = 2 (X X^{T} - \lambda) w
  $$
  For the general case of dimension $\geqslant 1$ $w$ is a set of vectors $w = (w\_1 w\_2 \ldots w\_n)$.
  This expression vanishes, if for some index $i$ $w\_i$ is an of eigenvector of $XX^{T}$ with the eigenvalue $\lambda\_i$, and all other components are set to zero. In other words, stationary points are the eigenvectors of $X X^{T}$.
  
  The condititon $w^T w = 1$ imposes the orthogonality condition on the eigenvectors. In fact, going back to the initial functional, one sees, that $w\_i X X^{T} w\_j = \lambda\_j w\_i^{T} w\_j = 0$ for $i \neq j$. Therefore, we have finally:
  $$
  \mathcal{L} =\sum \lambda\_i - \lambda
  $$
  Which is maximized for any $k \geq 1$, by taking $k$ largest eigenvalues.
  
  
  ---
  
  