Answers.Body: <p>for the first point I'm very sorry that I cannot give you any literature on this, but I might be able to explain you, why you don't take PCA on both datasets independently.</p>
  
  <p>Principal components analysis is simply a transformation of your data into another (less dimensional) coordinate system. The axis for your new coordinate system are defined by the principal components (i.e. eigen-vectors) of your covariance matrix.</p>
  
  <p>Since you will train your machine learning algorithm in the domain that is generated by the PCA, your test data must be exactly in the same domain. So as you said, you use exactly the same transformation for the test data as for the training data, i.e.
  <span class="math-container">$y_{newdim} = y \phi $</span>.
  Of course if you applied standardization to your training data, you have to apply the same standardization to your test data. So you need to store the mean <span class="math-container">$\mu_x$</span> and the standard deviation <span class="math-container">$S_x$</span> and also standardize your test data y to 
  <span class="math-container">\begin{equation}
  y_{standardized} = \dfrac{y_i - \mu_x}{S_x}
  \end{equation}</span></p>
  
  <p>Note here that you have an error in your standardization formula (you do not need to divide by <span class="math-container">$M$</span>).</p>
  
  <p>The point is that the principal components of your test data <span class="math-container">$\phi_{y}$</span> would not match the principal components of your training data <span class="math-container">$\phi_{x}$</span>. Thus the transformations from original space into PCA-space <span class="math-container">$\Phi_x(u)$</span> and <span class="math-container">$\Phi_y(u)$</span> would diverge and similar data points in the original space might be far away in the PCA-representation and vice versa. This is why you generate the mapping <span class="math-container">$x_{newdim} = \Phi_x(x) = x\phi$</span> and apply it also on the test data.</p>
  
  <p>I hope I could make it clear to you.</p>
  
  <p>Best</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-30T13:51:32.537Z
Answers.Text: Thanks @adriculteur. Than I do not need substract the zero-mean and std deviation that I did for the training set from the validation/test set? just multiply the validation/test dataset with the eigenvectors and it is done?
Answers.UserDisplayName: LVoltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-30T14:54:59.273Z
Answers.Text: Sorry if I had not been explicit on that. The test data has to undergo exactly the same transformation as the training data. I will edit my answer accordingly.
Answers.UserDisplayName: adriculteur
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-30T15:40:29.577Z
Answers.Text: sorry I wrote wrong, actually is srqt(M), as the article that I wrote as references, but it does not matter because it will cancel in the thrd equation. Anyway thank you so much for you aswer, it helps me a lot!!! Best regards!
Answers.UserDisplayName: LVoltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-30T15:47:27.623Z
Answers.Text: I am happy my answer helped you, but actually the standard score is calculated as you did in equation (3) but without the 1/sqrt(M). Just look at the equation in my answer or here: https://en.wikipedia.org/wiki/Standard_score#Calculation_from_raw_score
Answers.UserDisplayName: adriculteur
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-30T15:56:56.180Z
Answers.Text: Hi again! You are right, as I was following the article and the both equations have M or /sqrt(M) and then they call the covariance matrix as the correlation matrix when the difference between covariance matrix and correlation matrix is because the correlation matrix is a normalized form from covariance matrix. :)
Answers.UserDisplayName: LVoltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-30T13:46:16.000Z
Answers.LastActivityDate: 2018-10-30T15:02:56.687Z
Answers.LastEditDate: 2018-10-30T15:02:56.687Z
Answers.UserDisplayName: adriculteur
Body: <p>I was using PCA on my whole dataset (and, after that, I would split it into training, validation, and test datasets). However, after a little bit of research, I found out that this is the wrong way to do it.</p>
  <p>I have few questions:</p>
  <ol>
  <li><p>Are there some articles/references that explain why is the wrong way?</p>
  </li>
  <li><p>How can I transform the validation/test set?</p>
  </li>
  </ol>
  <p>Steps to do PCA (from <a href="https://www.sciencedirect.com/science/article/pii/S0022460X0093390X" rel="nofollow noreferrer">https://www.sciencedirect.com/science/article/pii/S0022460X0093390X</a>):</p>
  <ol>
  <li>zero mean</li>
  </ol>
  <p><span class="math-container">$$\mu = \frac{1}{M}\sum_{i=1}^{M} x_{i}$$</span></p>
  <p>where <strong>x</strong> is my training set</p>
  <ol start="2">
  <li>centering (variance)</li>
  </ol>
  <p><span class="math-container">$$S^{2} = \frac{1}{M}\sum_{i=1}^{M} (x_{i}-\mu)^{T}(x_{i}-\mu)$$</span></p>
  <ol start="3">
  <li>use (1) and (2) to transform my original training dataset</li>
  </ol>
  <p><span class="math-container">$$x_{new} = \frac{1}{\sqrt{M}} \frac{(x_{i} - \mu)}{S}$$</span></p>
  <ol start="4">
  <li>calculate covariance matrix (actually correlation matrix)</li>
  </ol>
  <p><span class="math-container">$$C= x_{new}^T x_{new}$$</span></p>
  <ol start="5">
  <li>take the <em>k</em>-eigenvectors (/phi) from the covariance matrix and defined the new space for my new dimension training set (where k are the principal components that I choose according to my variance)</li>
  </ol>
  <p><span class="math-container">$$ x_{new dim} = x_{new}\phi$$</span></p>
  <p>Ok, then I have my new dimensional training dataset after PCA (till here it's right, according to other papers that I have read).</p>
  <p>The question is: *<em>What I have to do now for my validation/testing set?</em> Just the equation below?</p>
  <p><span class="math-container">$$y_{new dim} = y\phi $$</span></p>
  <p>where <strong>y</strong> is my (for example) validation original dataset?</p>
  <p>Can someone explain the right thing to do?</p>
  
Tags: <machine-learning><principal-component-analysis><test-datasets><validation-datasets>
Title: How to perform PCA in the validation/test set?
UserDisplayName: LVoltz
ThreadText: # How to perform PCA in the validation/test set?
  
  (Asked by: username_0 on 2018-10-30)
  
  I was using PCA on my whole dataset (and, after that, I would split it into training, validation, and test datasets). However, after a little bit of research, I found out that this is the wrong way to do it.
  
  I have few questions:
  
  1. Are there some articles/references that explain why is the wrong way?
  2. How can I transform the validation/test set?
  
  Steps to do PCA (from <https://www.sciencedirect.com/science/article/pii/S0022460X0093390X>):
  
  1. zero mean
  
  $$\mu = \frac{1}{M}\sum\_{i=1}^{M} x\_{i}$$
  
  where **x** is my training set
  
  2. centering (variance)
  
  $$S^{2} = \frac{1}{M}\sum\_{i=1}^{M} (x\_{i}-\mu)^{T}(x\_{i}-\mu)$$
  
  3. use (1) and (2) to transform my original training dataset
  
  $$x\_{new} = \frac{1}{\sqrt{M}} \frac{(x\_{i} - \mu)}{S}$$
  
  4. calculate covariance matrix (actually correlation matrix)
  
  $$C= x\_{new}^T x\_{new}$$
  
  5. take the *k*-eigenvectors (/phi) from the covariance matrix and defined the new space for my new dimension training set (where k are the principal components that I choose according to my variance)
  
  $$ x\_{new dim} = x\_{new}\phi$$
  
  Ok, then I have my new dimensional training dataset after PCA (till here it's right, according to other papers that I have read).
  
  The question is: \**What I have to do now for my validation/testing set?* Just the equation below?
  
  $$y\_{new dim} = y\phi $$
  
  where **y** is my (for example) validation original dataset?
  
  Can someone explain the right thing to do?
  
  **Tags:** machine-learning, principal-component-analysis, test-datasets, validation-datasets
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-30. Score: 2 | Accepted answer
  
  for the first point I'm very sorry that I cannot give you any literature on this, but I might be able to explain you, why you don't take PCA on both datasets independently.
  
  Principal components analysis is simply a transformation of your data into another (less dimensional) coordinate system. The axis for your new coordinate system are defined by the principal components (i.e. eigen-vectors) of your covariance matrix.
  
  Since you will train your machine learning algorithm in the domain that is generated by the PCA, your test data must be exactly in the same domain. So as you said, you use exactly the same transformation for the test data as for the training data, i.e.
  $y\_{newdim} = y \phi $.
  Of course if you applied standardization to your training data, you have to apply the same standardization to your test data. So you need to store the mean $\mu\_x$ and the standard deviation $S\_x$ and also standardize your test data y to
  \begin{equation}
  y\_{standardized} = \dfrac{y\_i - \mu\_x}{S\_x}
  \end{equation}
  
  Note here that you have an error in your standardization formula (you do not need to divide by $M$).
  
  The point is that the principal components of your test data $\phi\_{y}$ would not match the principal components of your training data $\phi\_{x}$. Thus the transformations from original space into PCA-space $\Phi\_x(u)$ and $\Phi\_y(u)$ would diverge and similar data points in the original space might be far away in the PCA-representation and vice versa. This is why you generate the mapping $x\_{newdim} = \Phi\_x(x) = x\phi$ and apply it also on the test data.
  
  I hope I could make it clear to you.
  
  Best
  
  > **Comments:**
  > **username_0**: Thanks @adriculteur. Than I do not need substract the zero-mean and std deviation that I did for the training set from the validation/test set? just multiply the validation/test dataset with the eigenvectors and it is done?  
  > **username_1**: Sorry if I had not been explicit on that. The test data has to undergo exactly the same transformation as the training data. I will edit my answer accordingly.  
  > **username_0**: sorry I wrote wrong, actually is srqt(M), as the article that I wrote as references, but it does not matter because it will cancel in the thrd equation. Anyway thank you so much for you aswer, it helps me a lot!!! Best regards!  
  > **username_1**: I am happy my answer helped you, but actually the standard score is calculated as you did in equation (3) but without the 1/sqrt(M). Just look at the equation in my answer or here: https://en.wikipedia.org/wiki/Standard_score#Calculation_from_raw_score  
  > **username_0**: Hi again! You are right, as I was following the article and the both equations have M or /sqrt(M) and then they call the covariance matrix as the correlation matrix when the difference between covariance matrix and correlation matrix is because the correlation matrix is a normalized form from covariance matrix. :)  
  
  ---
  
  