Answers.Body: <p>Energy loss in principle component analysis (PCA) is the decrease in the sum of the squares of the coefficients of the scores obtained (so the coordinates for that row of a data set) from the covariance matrix relative to the sum of the squares of the coefficients of original input vectors (the true coordinates if you prefer to express it that way).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-16T16:19:00.347Z
Answers.Text: And what is the relation between your answer and the proposed way?
Answers.UserDisplayName: hasanghaforian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T03:45:37.280Z
Answers.Text: @hasanghaforian l don't understand your comment. You asked for a definition of data energy loss so I gave it. If by "the proposed way" you mean the wiki article then there are a few things you are not so sure of:
  The term "energy" is not defined in the article. I understand that they mean "Accumulated variance". A maximum likelihood estimator may be inside-out. For example the max of a mixture means splits the data of one component in two but the summed energy is smaller.
Answers.UserDisplayName: Pointwavestandard
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-16T05:43:48.093Z
Answers.LastActivityDate: 2024-01-16T05:43:48.093Z
Answers.UserDisplayName: Pointwavestandard
Body: <p>Recently in a slide in about <code>PCA</code> (Principal Component Analysis) I saw a question: &quot;How much is the data energy loss in <code>PCA</code>?&quot; Then in the next slide I saw somethings that do not understand relation of last steps to the <code>PCA</code> and energy. Although I know that <code>PCA</code> tries to find principle components which have variances in decreasing order and also guess the energy means the total variance after keeping some first principle components and applying it on normalized data; but in this way I think what is described in <a href="https://en.wikipedia.org/wiki/Principal_component_analysis" rel="nofollow noreferrer">Wikipedia</a> as <code>cumulative energy content for each eigenvector</code> and what is described <a href="https://stats.stackexchange.com/a/22571">here</a>, are more rational and understandable.</p>
  <p>Here you can see part of slide that tries to show an example:</p>
  <p><a href="https://i.sstatic.net/fUMJq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fUMJq.png" alt="enter image description here" /></a></p>
  <p>Steps are:</p>
  <ol>
  <li><code>X</code> as input</li>
  <li>normalize <code>X</code> (as <code>Z</code>)</li>
  <li>compute covariance matrix of <code>Z</code> as <code>R</code></li>
  <li>eigenvalues of <code>R</code> in descending order as <span class="math-container">$\lambda$</span> and their corresponding vectors as <code>V</code>.</li>
  <li><span class="math-container">$S = V\lambda^{\frac{1}{2}}$</span> where <span class="math-container">$\lambda^{\frac{1}{2}}$</span> is element-wise operation.</li>
  <li>discard third column of <code>S</code> and sore it as <code>S</code></li>
  <li>calculate diagonal elements of <span class="math-container">$S S^{T}$</span> as <code>var</code> columnar matrix</li>
  <li><code>1 - var(2,1)</code> is the amount of energy loss of <code>PCA</code>. Here it is <code>1 - 0.8420</code>.</li>
  </ol>
  <p>This is an <code>Octave</code> code that you can follow to get the same results:</p>
  <pre><code>X=[
  7 4 3;4 1 8;6 3 5;8 6 1;8 5 7;
  7 2 9;5 3 3;9 5 8;7 4 5;8 2 2];
  
  rows=size(X)(1);
  XMean = repmat(mean(X),rows,1);
  XStd = repmat(std(X),rows,1);
  X_normalized = (X - XMean)./(XStd);
  
  Cov_mat=cov(X_normalized);
  
  [V,lambda] = eigs(Cov_mat) %descending order
  S = V * lambda^(0.5);
  S1 = S(:,1:2);
  var = diag(S1*S1')
  </code></pre>
  <p>Here are what I do not understand:</p>
  <ul>
  <li>What is the definition of <code>energy loss</code> of PCA? If it is relative to what is described in <a href="https://en.wikipedia.org/wiki/Principal_component_analysis" rel="nofollow noreferrer">Wikipedia</a>, then how is it relative to the steps that here are described in above slide?</li>
  <li>Where can I found a reference or some examples about these steps?</li>
  <li>Why did we keep only first two columns of <code>S</code>?</li>
  <li>Why <code>1 - var(2,1)</code> is the amount of energy loss of <code>PCA</code>? Does <code>2</code> in <code>1 - var(2,1)</code>
  is relative to the first <em>two</em> columns?</li>
  <li>Is <code>1 - var(2,1)</code> is the amount of energy loss of <code>PCA</code> by using two eigenvectors relative to the two eigenvectors with largest magnitude?</li>
  <li>If we want to calculate energy loss for only using eigenvector that is relative to the largest eigenvalue, do we have to calculate <span class="math-container">$[S(1,1) S(1,2) S(1,3)]^T [S(1,1) S(1,2) S(1,3)]$</span> as <code>var</code> and <code>1 - var(1,1)</code> is the amount of energy loss?</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-17T01:03:32.647Z
Comments.Text: Please, split this post into multiple ones. Here you're asking multiple questions in the same post, which is discouraged because people are discouraged to address your post as it can become too complicated. So, ideally, you should ask one question per post.
Comments.UserDisplayName: nbro
Tags: <machine-learning><math><feature-extraction><statistical-ai><principal-component-analysis>
Title: What is the meaning of "energy loss" in PCA?
UserDisplayName: hasanghaforian
ThreadText: # What is the meaning of "energy loss" in PCA?
  
  (Asked by: username_0 on 2024-01-16)
  
  Recently in a slide in about PCA (Principal Component Analysis) I saw a question: "How much is the data energy loss in PCA?" Then in the next slide I saw somethings that do not understand relation of last steps to the PCA and energy. Although I know that PCA tries to find principle components which have variances in decreasing order and also guess the energy means the total variance after keeping some first principle components and applying it on normalized data; but in this way I think what is described in Wikipedia as cumulative energy content for each eigenvector and what is described here, are more rational and understandable.
  Here you can see part of slide that tries to show an example:
  
  Steps are:
  
  X as input
  normalize X (as Z)
  compute covariance matrix of Z as R
  eigenvalues of R in descending order as $\lambda$ and their corresponding vectors as V.
  $S = V\lambda^{\frac{1}{2}}$ where $\lambda^{\frac{1}{2}}$ is element-wise operation.
  discard third column of S and sore it as S
  calculate diagonal elements of $S S^{T}$ as var columnar matrix
  1 - var(2,1) is the amount of energy loss of PCA. Here it is 1 - 0.8420.
  
  This is an Octave code that you can follow to get the same results:
  X=[
  7 4 3;4 1 8;6 3 5;8 6 1;8 5 7;
  7 2 9;5 3 3;9 5 8;7 4 5;8 2 2];
  
  rows=size(X)(1);
  XMean = repmat(mean(X),rows,1);
  XStd = repmat(std(X),rows,1);
  X_normalized = (X - XMean)./(XStd);
  
  Cov_mat=cov(X_normalized);
  
  [V,lambda] = eigs(Cov_mat) %descending order
  S = V * lambda^(0.5);
  S1 = S(:,1:2);
  var = diag(S1*S1')
  
  Here are what I do not understand:
  
  What is the definition of energy loss of PCA? If it is relative to what is described in Wikipedia, then how is it relative to the steps that here are described in above slide?
  Where can I found a reference or some examples about these steps?
  Why did we keep only first two columns of S?
  Why 1 - var(2,1) is the amount of energy loss of PCA? Does 2 in 1 - var(2,1)
  is relative to the first two columns?
  Is 1 - var(2,1) is the amount of energy loss of PCA by using two eigenvectors relative to the two eigenvectors with largest magnitude?
  If we want to calculate energy loss for only using eigenvector that is relative to the largest eigenvalue, do we have to calculate $[S(1,1) S(1,2) S(1,3)]^T [S(1,1) S(1,2) S(1,3)]$ as var and 1 - var(1,1) is the amount of energy loss?
  
  **Tags:** machine-learning, math, feature-extraction, statistical-ai, principal-component-analysis
  > **Comments:**
  > **username_1**: Please, split this post into multiple ones. Here you're asking multiple questions in the same post, which is discouraged because people are discouraged to address your post as it can become too complicated. So, ideally, you should ask one question per post.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-01-16. Score: -1
  
  Energy loss in principle component analysis (PCA) is the decrease in the sum of the squares of the coefficients of the scores obtained (so the coordinates for that row of a data set) from the covariance matrix relative to the sum of the squares of the coefficients of original input vectors (the true coordinates if you prefer to express it that way).
  
  > **Comments:**
  > **username_0**: And what is the relation between your answer and the proposed way?  
  > **username_2**: @hasanghaforian l don't understand your comment. You asked for a definition of data energy loss so I gave it. If by "the proposed way" you mean the wiki article then there are a few things you are not so sure of: The term "energy" is not defined in the article. I understand that they mean "Accumulated variance". A maximum likelihood estimator may be inside-out. For example the max of a mixture means splits the data of one component in two but the summed energy is smaller.  
  
  ---
  
  