Answers.Body: <h3>The empirical statistical point of view</h3>
  <p>The technique is  uniform manifold approximation and projection (UMAP) and is comparable to t-SNE. UMAP is considered a slight improvement, personally I think they have similar behaviour. They are dimensionality reduction methods.</p>
  <p>The reason UMAP or t-SNE is often not applied directly is it is weaker than PCA are capturing the multi-dimensional variance of the data. Both methods can be applied directly to the data, but will not necessarily deliver stronger clusters than PCA (which is the goal of the technique).</p>
  <p>However, the weakness of PCA is that the variance can be spread across 50 principle components, making it hard to recover the representative clusters in certain situations. These situations are commonly found in RNA expression analytics, notably where the first 3 or 4 PCs only comprise ~30% of the variance. PCA should be trapping &gt;80% of the variance within the first 3 or 4 PCs. In expression data (RNAseq) the next 46 PCs might comprise a total of ~30% of the variance and omitting that is mistake, but manually comparing 50 PCs in bivariate plots is not practical. So at best a good unsupervised learning algorithm (UMAP/t-SNE) following PCA in RNA expression via RNAseq might trap 60% of the variance. It's not ideal but it's okay. Thus you can't ignore the 30% of the variance in the bottom 46 PCs and UMAP/t-SNE is recovering that. If 80% of the variance was in the top 4 PCs, then the remaining 20% of the variance in the lower 46 PCs can be ignored and t-SNE or UMAP is not needed.</p>
  <p><strong>Resolution</strong> The way you can capture the variance from all 50 PCs is to perform a clustering method over the top of PCA, this being t-SNE or UMAP.</p>
  <p>Summary,</p>
  <ul>
  <li>t-SNE or UMAP without PCA might trap at best 20% of the variance.</li>
  <li>The first 4 PCs in a PCA might trap 30% of the variance (it will be a better result than t-SNE or UMAP).</li>
  <li>Combining PCA and then UMAP/t-SNE will capture 60% of the variance.</li>
  </ul>
  <p>60% of the variance, its just about acceptable. Thus thats why this approach  is done from an empirical point of view.</p>
  <h3>Practical point of view</h3>
  <p>PCA and then UMAP (or t-SNE) will give tighter, neater clusters (cleaner, less messy groups), but thats not the reason why you are using it.</p>
  <hr />
  <p>The publications in the link,</p>
  <p><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10312943/" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10312943/</a>
  and the PLoS Biology paper
  <a href="https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001532" rel="nofollow noreferrer">https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001532</a></p>
  <p>... both use UMAP across PCA. Its a major approach to cluster resolution.</p>
  <p>Diffusion maps are an attempt to extrapolate a variance within Eigenvalues via a random walk, i.e. to defuse within Euclidean space. It is not a primary calculation in my personal opinion because you are sort of introducing more variance into a variance dimensionality reduction calculation. Once you have very tight clusters using PCA/UMAP or PCA/tSNE then you could check to see whether Diffusion map clusters are similar.</p>
  <p>This is my perspective only, not to replace PCA with Diffusion map nor to consider a &quot;down-stream&quot; calculation. Its separate calculation used to verify if PCA-UMAP is okay.</p>
  <p>If you doing graph theory a Diffusion map could be really important as a null hypothesis and a really important way to obtain a different understanding of the graph (i.e. this is a technical term for a network). Using a diffusion map without any directionality (the papers you cited)*, I struggle to see this as a primary analysis. In the PLOS Biology paper is part of the appendix.</p>
  <p>*, graph says A-&gt;B-&gt;C
  The publications treat A,B, C as unconnected. If you can hypotheses how A, B and C are connected then diffusion map could be cool. Thats my opinion from the perspective of graph theory.</p>
  <p><strong>Summary</strong> I would definitely choose PCA-UMAP over a diffusion map every time. This is what has happened in the two publications cited. The diffusion map is secondary analysis, so by all means do it ... but present PCA-UMAP first, and base the results around that.</p>
  <hr />
  <p>From the comments there's some sort of time-series. So if you are performing a time-series - it's not datamining - this would be the justification of using a Diffusion map over PCA. This therefore answers points 1 and 2. However, it would imply you are pooling your data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-17T16:03:30.993Z
Answers.Text: The specific technique is a Diffusion Map after PCA (and not UMAP/t-SNE). Does the same reasoning hold for the Diffusion Map dimensionality reduction as well?
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-17T18:15:53.203Z
Answers.Text: Thanks for clarification, I've answered above
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-18T09:30:12.213Z
Answers.Text: I'll explain my specific case in more detail : The reason I personally use a Diffusion Map is not only to see the clusters, but to draw a trajectory between the clusters. In my case, not only the Diffusion Map separates better between the (two) main clusters, but the genes change in a meaningful way, across the trajectory : from state A, to intermediate state, to state B.  However, this is not stable in the sense that it appears only when using the normalized data as a basis for Diffusion Map, not when using the 50 PCs.Hence, the two OP questions focus on Diffusion Map(on normalized vs on PCs)
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-18T14:21:38.470Z
Answers.Text: @Sam maybe this is a separate question? The studies you presented suggested a straight cluster analysis. If you are doing e.g. a time series that will require a lot more thought.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-18T15:21:34.700Z
Answers.Text: I have intended the studies just to illustrate the point that taking the first 50 PCs is often done (so that I do not get called out for making unfounded statements about what is often done). They were not intended to be the focus of the question, but I see it was not clear.
  
  How would you phrase a separate question? In my mind, the question is well-phrased (but it might be I don't get it).
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-18T16:14:42.740Z
Answers.Text: Are you doing a time-series?
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-19T09:24:34.057Z
Answers.Text: My data has been collected at one time-point, but computationally I perform trajectory analysis using Slingshot, based on that, divide the trajectory into 3 timepoints (start, mid, end), and perform differential expression test between the timepoints using tradeSeq
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T09:51:43.143Z
Answers.Text: can you help further?
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T14:44:53.643Z
Answers.Text: Thanks @Sam answered above
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-14T22:56:11.740Z
Answers.LastActivityDate: 2024-01-02T14:44:36.410Z
Answers.LastEditDate: 2024-01-02T14:44:36.410Z
Answers.UserDisplayName: M__
Body: <p><em>This question was also asked on <a href="https://github.com/theislab/destiny/issues/64" rel="nofollow noreferrer">GitHub</a></em></p>
  <p>In the R package destiny, a Diffusion Map is <a href="https://scholar.google.com/scholar?hl=en&amp;as_sdt=0%2C5&amp;q=%22n_pcs%3D50%22%20%20%20destiny&amp;btnG=" rel="nofollow noreferrer">often</a> performed on the first 50 PCs of the data.</p>
  <ol>
  <li><p>Is there a theoretical reason for performing a Diffusion Map on PCs rather than on the data itself ( other than speed)?</p>
  </li>
  <li><p>Is it reasonable to do the diffusion map on the normalized data itself rather than on Principal Components of the data, if that differentiates the states better ?</p>
  </li>
  </ol>
  <hr />
  <p>Techinically, I am using the  R Bioconductor package destiny. The help page on destiny::DiffusionMap tells &quot;Using e.g. 50 PCs results in more regular looking diffusion maps&quot;, but it's not clear compared to what will they be more regularly looking.</p>
  
Tags: <dimensionality-reduction>
Title: Using Diffusion Map on normalized - or PCA reduced data?
ThreadText: # Using Diffusion Map on normalized - or PCA reduced data?
  
  (Asked by: username_0 on 2023-12-10)
  
  *This question was also asked on [GitHub](https://github.com/theislab/destiny/issues/64)*
  
  In the R package destiny, a Diffusion Map is [often](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22n_pcs%3D50%22%20%20%20destiny&btnG=) performed on the first 50 PCs of the data.
  
  1. Is there a theoretical reason for performing a Diffusion Map on PCs rather than on the data itself ( other than speed)?
  2. Is it reasonable to do the diffusion map on the normalized data itself rather than on Principal Components of the data, if that differentiates the states better ?
  
  ---
  
  Techinically, I am using the R Bioconductor package destiny. The help page on destiny::DiffusionMap tells "Using e.g. 50 PCs results in more regular looking diffusion maps", but it's not clear compared to what will they be more regularly looking.
  
  **Tags:** dimensionality-reduction
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-14. Score: 2
  
  ### The empirical statistical point of view
  
  The technique is uniform manifold approximation and projection (UMAP) and is comparable to t-SNE. UMAP is considered a slight improvement, personally I think they have similar behaviour. They are dimensionality reduction methods.
  
  The reason UMAP or t-SNE is often not applied directly is it is weaker than PCA are capturing the multi-dimensional variance of the data. Both methods can be applied directly to the data, but will not necessarily deliver stronger clusters than PCA (which is the goal of the technique).
  
  However, the weakness of PCA is that the variance can be spread across 50 principle components, making it hard to recover the representative clusters in certain situations. These situations are commonly found in RNA expression analytics, notably where the first 3 or 4 PCs only comprise ~30% of the variance. PCA should be trapping >80% of the variance within the first 3 or 4 PCs. In expression data (RNAseq) the next 46 PCs might comprise a total of ~30% of the variance and omitting that is mistake, but manually comparing 50 PCs in bivariate plots is not practical. So at best a good unsupervised learning algorithm (UMAP/t-SNE) following PCA in RNA expression via RNAseq might trap 60% of the variance. It's not ideal but it's okay. Thus you can't ignore the 30% of the variance in the bottom 46 PCs and UMAP/t-SNE is recovering that. If 80% of the variance was in the top 4 PCs, then the remaining 20% of the variance in the lower 46 PCs can be ignored and t-SNE or UMAP is not needed.
  
  **Resolution** The way you can capture the variance from all 50 PCs is to perform a clustering method over the top of PCA, this being t-SNE or UMAP.
  
  Summary,
  
  * t-SNE or UMAP without PCA might trap at best 20% of the variance.
  * The first 4 PCs in a PCA might trap 30% of the variance (it will be a better result than t-SNE or UMAP).
  * Combining PCA and then UMAP/t-SNE will capture 60% of the variance.
  
  60% of the variance, its just about acceptable. Thus thats why this approach is done from an empirical point of view.
  
  ### Practical point of view
  
  PCA and then UMAP (or t-SNE) will give tighter, neater clusters (cleaner, less messy groups), but thats not the reason why you are using it.
  
  ---
  
  The publications in the link,
  
  <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10312943/>
  and the PLoS Biology paper
  <https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001532>
  
  ... both use UMAP across PCA. Its a major approach to cluster resolution.
  
  Diffusion maps are an attempt to extrapolate a variance within Eigenvalues via a random walk, i.e. to defuse within Euclidean space. It is not a primary calculation in my personal opinion because you are sort of introducing more variance into a variance dimensionality reduction calculation. Once you have very tight clusters using PCA/UMAP or PCA/tSNE then you could check to see whether Diffusion map clusters are similar.
  
  This is my perspective only, not to replace PCA with Diffusion map nor to consider a "down-stream" calculation. Its separate calculation used to verify if PCA-UMAP is okay.
  
  If you doing graph theory a Diffusion map could be really important as a null hypothesis and a really important way to obtain a different understanding of the graph (i.e. this is a technical term for a network). Using a diffusion map without any directionality (the papers you cited)\*, I struggle to see this as a primary analysis. In the PLOS Biology paper is part of the appendix.
  
  \*, graph says A->B->C
  The publications treat A,B, C as unconnected. If you can hypotheses how A, B and C are connected then diffusion map could be cool. Thats my opinion from the perspective of graph theory.
  
  **Summary** I would definitely choose PCA-UMAP over a diffusion map every time. This is what has happened in the two publications cited. The diffusion map is secondary analysis, so by all means do it ... but present PCA-UMAP first, and base the results around that.
  
  ---
  
  From the comments there's some sort of time-series. So if you are performing a time-series - it's not datamining - this would be the justification of using a Diffusion map over PCA. This therefore answers points 1 and 2. However, it would imply you are pooling your data.
  
  > **Comments:**
  > **username_0**: The specific technique is a Diffusion Map after PCA (and not UMAP/t-SNE). Does the same reasoning hold for the Diffusion Map dimensionality reduction as well?  
  > **username_1**: Thanks for clarification, I've answered above  
  > **username_0**: I'll explain my specific case in more detail : The reason I personally use a Diffusion Map is not only to see the clusters, but to draw a trajectory between the clusters. In my case, not only the Diffusion Map separates better between the (two) main clusters, but the genes change in a meaningful way, across the trajectory : from state A, to intermediate state, to state B.  However, this is not stable in the sense that it appears only when using the normalized data as a basis for Diffusion Map, not when using the 50 PCs.Hence, the two OP questions focus on Diffusion Map(on normalized vs on PCs)  
  > **username_1**: @username_0 maybe this is a separate question? The studies you presented suggested a straight cluster analysis. If you are doing e.g. a time series that will require a lot more thought.  
  > **username_0**: I have intended the studies just to illustrate the point that taking the first 50 PCs is often done (so that I do not get called out for making unfounded statements about what is often done). They were not intended to be the focus of the question, but I see it was not clear.  How would you phrase a separate question? In my mind, the question is well-phrased (but it might be I don't get it).  
  > **username_1**: Are you doing a time-series?  
  > **username_0**: My data has been collected at one time-point, but computationally I perform trajectory analysis using Slingshot, based on that, divide the trajectory into 3 timepoints (start, mid, end), and perform differential expression test between the timepoints using tradeSeq  
  > **username_0**: can you help further?  
  > **username_1**: Thanks @username_0 answered above  
  
  ---
  
  