Body: <p>I've got a moderately large set of PBMCs, over 1M cells. That means I can't easily do a grid search of dimensionality reduction/clustering parameters/methods. Some examples results I'm getting with scanpy default parameters include PCA -- a big, dense grouping with lobes radiating outwards, and UMAP -- &quot;better&quot; groupings in some sense, but many many tiny clusters of cells, more than are biologically plausible (hundreds) as well.</p>
  <p>Basically I'm looking for best dimred/clustering practices for such a dataset, other than what's on offer from scanpy/Seurat. My metrics such as they are are that the major cell types should be clearly separated, and that the overall plot have some biological plausibility in terms of number of visually distinct clusters.</p>
  <p>I'm aware of the Xiang et al Frontier Genetics 2021 paper, but the number of cells their example datasets use is much smaller AFAIK. They concluded that tSNE is the best choice of method according to their metrics, though UMAP also has some stability advantages. The Chari and Pachter PLOS Comp. Bio. 2023 paper raises some good points (a highly dimensional space can't really accurately have its relations preserved in 2 dimensions, and there are other issues of noise, interpretability, and so on. They recommend clustering from the higher dimensional data, which I do agree with. A different paper rec could be great.</p>
  <p>PS: SCVI seems like a good option, I'm playing with that at the moment.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-25T22:36:14.497Z
Comments.Text: Hi M, thanks for the reply. The first paper I mentioned covers several dimensionality reduction methods and concludes that for the older data they tested, tSNE is best. The second one covers some of the theory and recommends perhaps side stepping dimred entirely.
Comments.UserDisplayName: Henry Gong
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-25T22:38:32.050Z
Comments.Text: As for metric, basically I'm looking for separation of major cell types and a plausible overall number of clusters. PCA for instance does separate some PBMC cell types out but not most. UMAP does separate out certain cell types (B cells are well separated for instance) but not others and seems to have lots of spurious tiny clusters.
Comments.UserDisplayName: Henry Gong
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-25T22:56:14.130Z
Comments.Text: Yes, I'll edit it in a bit -- will write up a less pithy short summary of the second paper.
Comments.UserDisplayName: Henry Gong
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-26T15:42:46.893Z
Comments.Text: I always wonder what 1M cells can answer what 100k cannot. Anyway, what you have to find out based on published datasets of that magnitude is which approaches scale to that number of cells. I never had to do that myself since our datasets are always way smaller. I would consider not to analyse this per-cell but rather try to find some sort of meta-cell approach where you bin cells into metas, to reduce the computational burden. I would not believe that with 1M cells one really benefits from pure single-cell analysis. Maybe some sort of clustering to get "pseudobulks" and then use these.
Comments.UserDisplayName: ATpoint
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-26T22:18:53.030Z
Comments.Text: @M, just SCVI at the moment. @ATpoint, it's 1M+ cells across many individuals, and I do agree that cluster level comparisons are a good way to make the data tractable -- hence the question, really. Point well taken to compare to other similar datasets.
Comments.UserDisplayName: Henry Gong
Tags: <rna-seq><single-cell><clustering><pca><dimensionality-reduction>
Title: scRNA: What are good dimensionality reduction/clustering parameters to get biologically plausible groupings?
ThreadText: # scRNA: What are good dimensionality reduction/clustering parameters to get biologically plausible groupings?
  
  (Asked by: username_0 on 2023-08-25)
  
  I've got a moderately large set of PBMCs, over 1M cells. That means I can't easily do a grid search of dimensionality reduction/clustering parameters/methods. Some examples results I'm getting with scanpy default parameters include PCA -- a big, dense grouping with lobes radiating outwards, and UMAP -- "better" groupings in some sense, but many many tiny clusters of cells, more than are biologically plausible (hundreds) as well.
  
  Basically I'm looking for best dimred/clustering practices for such a dataset, other than what's on offer from scanpy/Seurat. My metrics such as they are are that the major cell types should be clearly separated, and that the overall plot have some biological plausibility in terms of number of visually distinct clusters.
  
  I'm aware of the Xiang et al Frontier Genetics 2021 paper, but the number of cells their example datasets use is much smaller AFAIK. They concluded that tSNE is the best choice of method according to their metrics, though UMAP also has some stability advantages. The Chari and Pachter PLOS Comp. Bio. 2023 paper raises some good points (a highly dimensional space can't really accurately have its relations preserved in 2 dimensions, and there are other issues of noise, interpretability, and so on. They recommend clustering from the higher dimensional data, which I do agree with. A different paper rec could be great.
  
  PS: SCVI seems like a good option, I'm playing with that at the moment.
  
  **Tags:** rna-seq, single-cell, clustering, pca, dimensionality-reduction
  
  > **Comments:**
  > **username_0**: Hi M, thanks for the reply. The first paper I mentioned covers several dimensionality reduction methods and concludes that for the older data they tested, tSNE is best. The second one covers some of the theory and recommends perhaps side stepping dimred entirely.  
  > **username_0**: As for metric, basically I'm looking for separation of major cell types and a plausible overall number of clusters. PCA for instance does separate some PBMC cell types out but not most. UMAP does separate out certain cell types (B cells are well separated for instance) but not others and seems to have lots of spurious tiny clusters.  
  > **username_0**: Yes, I'll edit it in a bit -- will write up a less pithy short summary of the second paper.  
  > **username_1**: I always wonder what 1M cells can answer what 100k cannot. Anyway, what you have to find out based on published datasets of that magnitude is which approaches scale to that number of cells. I never had to do that myself since our datasets are always way smaller. I would consider not to analyse this per-cell but rather try to find some sort of meta-cell approach where you bin cells into metas, to reduce the computational burden. I would not believe that with 1M cells one really benefits from pure single-cell analysis. Maybe some sort of clustering to get "pseudobulks" and then use these.  
  > **username_0**: @M, just SCVI at the moment. @username_1, it's 1M+ cells across many individuals, and I do agree that cluster level comparisons are a good way to make the data tractable -- hence the question, really. Point well taken to compare to other similar datasets.  
  
  ---
  
  ## 0 answers
  
  