Answers.Body: <p>It really depends on what you want to learn from your data, which you haven't really explained yet.</p>
  <p>You have talked about explained variance but this is probably one of the less informative/useful outputs of PCA, so I doubt that it is why you did scRNA-seq. kNN/UMAP could be applied to each dataset individually or to all datasets at the same time, and it's not clear which you want.</p>
  <p>For example: <strong>do you want your different PCA results (patients) to be comparable?</strong> If so, then you should perform the PCA on everything all at once. That is the only way to ensure that the axes of the different experiments represent shared covariance.</p>
  <p>If you don't care about the comparability of your different scRNA-seq datasets (patients), then go ahead and analyze them separately. They will be easier to interpret since the batch effects will probably dominate in the larger grouping.</p>
  <p>However, if you want to learn something general across your patients in the PCA, then you will need to combine everything so that you account for the batch effects. Otherwise there is no guarantee that the covariance matrices that are the basis for your decomposition can be decomposed into PCs that are even slightly related to one another. It isn't like there's some underlying truth that PCA is finding in each experiment, it's merely whatever structure is present in the data.</p>
  <p>Granted, it's very possible that <em>qualitatively</em> you can learn something generalizable by examining the results of each patient independently. It just won't be something quantitatively discovered in your analysis.</p>
  <p>caveat: not a scRNA-seq person.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-28T03:13:36.723Z
Answers.Text: I'm definitely agreed on all points of the post.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-31T17:22:46.017Z
Answers.Text: I edited my post to make it more precise.
  But your answer is helpful for my case, but also for other ones that are interested in analyzing any data.
Answers.UserDisplayName: Wassim Jaoui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-28T01:57:45.640Z
Answers.LastActivityDate: 2024-12-28T01:57:45.640Z
Answers.UserDisplayName: Maximilian Press
Answers.Body: <p>UMAP is a strong clustering algorithm. KNN is used in a lot of unsupervised learning, generally outside life-science. KNN is very cool but often gets replaced by other algorithms, here it would be UMAP, in other walks of life its cosine similarity.</p>
  <p>The issue with PCA and ANY RNAseq method is that the variance gets &quot;smashed&quot; (best word I can think of) amongst A LOT of principle components. The idea that 80% of the variance is trapped within the first 3 principle components (which is classical statistics), just doesn't happen in RNAseq. You'd be lucky to get 25% of the variance in the first three PCs. The variance can be spread over 50+ PCs.</p>
  <ul>
  <li>if you are using PCA I'd recommend you assess the variance trapped per PC empirically</li>
  </ul>
  <p>Thus one approach is to do a PCA and then perform a second clustering over all (or a large number) of the PCs, tSNE is often used for this.</p>
  <p>There is an issue about using patient data and standardisation (previous post) and thats why I'd personally examine each individual PCA plot before attempting to do a single plot for all data. This is because the overlapping clusters can fail due to variation in patient sampling. Like if each individual PCA plot shows very similar clusters and THEN the single PCA plots shows &quot;chaos&quot;, there's an issue about standardisation and that can be complicated to sort out.</p>
  <p>The ultimate issue about any unsupervised learning method is as an end point analysis, there's no real hard and fast rules. Basically, the rule quite literally is, if it produces nice clusters you are good to go. I mean literally thats the criteria, it's the only branch of stats I know where the &quot;nice result&quot; is a justification of the approach. At least when you build up individual PCAs before diving into the big final analysis of &quot;the one PCA&quot; there's a process that can justify or &quot;debug&quot; the final thing.</p>
  <p>Dimensionality however can be a very cool statistical trick, but often it is seen as hypothesis building outside scRNA analysis.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-31T17:29:33.397Z
Answers.Text: Indeed, after filtering the scRNA data to the 100 genes with very high variance (using Seurat preprocessing techniques), I nearly get 1:1 growth in variance ratio with respect to the number of components in this range. That is, all genes are important, and they can barely be smashed into a lower dimension with PCA.
Answers.UserDisplayName: Wassim Jaoui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-31T18:33:11.373Z
Answers.Text: In that case @WassimJaoui if you use a PC1 versus PC2 plot that should provide a strong clustering and thats your answer. There is no need to go beyond that. If your 100 loci have been tightly selected - its possible
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-28T03:00:59.887Z
Answers.LastActivityDate: 2024-12-28T03:12:47.257Z
Answers.LastEditDate: 2024-12-28T03:12:47.257Z
Answers.UserDisplayName: M__
Body: <p>I am working with Single-Cell RNA-seq count matrices (gene expression x cells) from multiple files, which represent different samples and patients. I want to perform dimensionality reduction using PCA <strong>for visualization and clustering</strong> (UMAP and KNN).</p>
  <p>My question is:</p>
  <p>Which approach is better for consistency across datasets and preserving of important structure?</p>
  <p>Apply PCA on the combined dataset: Should I combine all the files into one dataset and then apply PCA on the entire dataset, check the explained variance ratio (EVR), and save the projection learned from the combined data? I would then apply this projection to each individual file, using the same number of components. This approach ensures that I am working with the same components across all files, making the datasets more comparable and preserving global structure.</p>
  <p>Apply PCA on each file separately: Alternatively, should I apply PCA on each file individually and then average the explained variance ratios (EVRs)? While this approach allows for the dimensionality reduction of each file separately, it may result in different components across files, making the datasets less comparable and possibly losing global structure.</p>
  <p>Which approach is more suitable for preserving global structure, ensuring comparability, and maintaining consistency in dimensionality reduction across multiple scRNA-seq datasets?</p>
  <p>I'm working in a project where my ML algorithm tries to learn a mapping between the control and treated patients' gene expression. For that it will have to compare multiple samples with each other, before and after the treatment and try to understand the function behind each type of therapy. Therefore the project necessitates the conservation of global structure and that the preprocessing of the data doesn't hinder a comparison of multiple samples.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-27T18:11:46.567Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Tags: <scrnaseq><visualization><machine-learning><pca><data-preprocessing>
Title: Should I apply PCA on the combined scRNA dataset or on each individual file?
ThreadText: # Should I apply PCA on the combined scRNA dataset or on each individual file?
  
  (Asked by: username_0 on 2024-12-27)
  
  I am working with Single-Cell RNA-seq count matrices (gene expression x cells) from multiple files, which represent different samples and patients. I want to perform dimensionality reduction using PCA **for visualization and clustering** (UMAP and KNN).
  
  My question is:
  
  Which approach is better for consistency across datasets and preserving of important structure?
  
  Apply PCA on the combined dataset: Should I combine all the files into one dataset and then apply PCA on the entire dataset, check the explained variance ratio (EVR), and save the projection learned from the combined data? I would then apply this projection to each individual file, using the same number of components. This approach ensures that I am working with the same components across all files, making the datasets more comparable and preserving global structure.
  
  Apply PCA on each file separately: Alternatively, should I apply PCA on each file individually and then average the explained variance ratios (EVRs)? While this approach allows for the dimensionality reduction of each file separately, it may result in different components across files, making the datasets less comparable and possibly losing global structure.
  
  Which approach is more suitable for preserving global structure, ensuring comparability, and maintaining consistency in dimensionality reduction across multiple scRNA-seq datasets?
  
  I'm working in a project where my ML algorithm tries to learn a mapping between the control and treated patients' gene expression. For that it will have to compare multiple samples with each other, before and after the treatment and try to understand the function behind each type of therapy. Therefore the project necessitates the conservation of global structure and that the preprocessing of the data doesn't hinder a comparison of multiple samples.
  
  **Tags:** scrnaseq, visualization, machine-learning, pca, data-preprocessing
  
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-12-28. Score: 2
  
  It really depends on what you want to learn from your data, which you haven't really explained yet.
  
  You have talked about explained variance but this is probably one of the less informative/useful outputs of PCA, so I doubt that it is why you did scRNA-seq. kNN/UMAP could be applied to each dataset individually or to all datasets at the same time, and it's not clear which you want.
  
  For example: **do you want your different PCA results (patients) to be comparable?** If so, then you should perform the PCA on everything all at once. That is the only way to ensure that the axes of the different experiments represent shared covariance.
  
  If you don't care about the comparability of your different scRNA-seq datasets (patients), then go ahead and analyze them separately. They will be easier to interpret since the batch effects will probably dominate in the larger grouping.
  
  However, if you want to learn something general across your patients in the PCA, then you will need to combine everything so that you account for the batch effects. Otherwise there is no guarantee that the covariance matrices that are the basis for your decomposition can be decomposed into PCs that are even slightly related to one another. It isn't like there's some underlying truth that PCA is finding in each experiment, it's merely whatever structure is present in the data.
  
  Granted, it's very possible that *qualitatively* you can learn something generalizable by examining the results of each patient independently. It just won't be something quantitatively discovered in your analysis.
  
  caveat: not a scRNA-seq person.
  
  > **Comments:**
  > **username_3**: I'm definitely agreed on all points of the post.  
  > **username_0**: I edited my post to make it more precise. But your answer is helpful for my case, but also for other ones that are interested in analyzing any data.  
  
  ---
  
  ### Answer by username_3 on 2024-12-28. Score: 1
  
  UMAP is a strong clustering algorithm. KNN is used in a lot of unsupervised learning, generally outside life-science. KNN is very cool but often gets replaced by other algorithms, here it would be UMAP, in other walks of life its cosine similarity.
  
  The issue with PCA and ANY RNAseq method is that the variance gets "smashed" (best word I can think of) amongst A LOT of principle components. The idea that 80% of the variance is trapped within the first 3 principle components (which is classical statistics), just doesn't happen in RNAseq. You'd be lucky to get 25% of the variance in the first three PCs. The variance can be spread over 50+ PCs.
  
  * if you are using PCA I'd recommend you assess the variance trapped per PC empirically
  
  Thus one approach is to do a PCA and then perform a second clustering over all (or a large number) of the PCs, tSNE is often used for this.
  
  There is an issue about using patient data and standardisation (previous post) and thats why I'd personally examine each individual PCA plot before attempting to do a single plot for all data. This is because the overlapping clusters can fail due to variation in patient sampling. Like if each individual PCA plot shows very similar clusters and THEN the single PCA plots shows "chaos", there's an issue about standardisation and that can be complicated to sort out.
  
  The ultimate issue about any unsupervised learning method is as an end point analysis, there's no real hard and fast rules. Basically, the rule quite literally is, if it produces nice clusters you are good to go. I mean literally thats the criteria, it's the only branch of stats I know where the "nice result" is a justification of the approach. At least when you build up individual PCAs before diving into the big final analysis of "the one PCA" there's a process that can justify or "debug" the final thing.
  
  Dimensionality however can be a very cool statistical trick, but often it is seen as hypothesis building outside scRNA analysis.
  
  > **Comments:**
  > **username_0**: Indeed, after filtering the scRNA data to the 100 genes with very high variance (using Seurat preprocessing techniques), I nearly get 1:1 growth in variance ratio with respect to the number of components in this range. That is, all genes are important, and they can barely be smashed into a lower dimension with PCA.  
  > **username_3**: In that case @username_0 if you use a PC1 versus PC2 plot that should provide a strong clustering and thats your answer. There is no need to go beyond that. If your 100 loci have been tightly selected - its possible  
  
  ---
  
  