Answers.Body: <p><em>[pasting my Reddit answer here]</em></p>
  <p>You don't. UMAP is a data visualisation technique, not a clustering technique. UMAP has little function beyond providing a way to plot thousands of multi-dimensional data points in an aesthetically-pleasing way.</p>
  <p>What you're asking is like asking how to quantitatively evaluate the layout of a histogram.</p>
  <hr />
  <p>If, on the other hand, you're asking about <em>clustering</em> rather than <em>visualisation</em>, then Daniela Witten's group has developed something they call Data Thinning, which may be of use to you:</p>
  <ul>
  <li><a href="https://www.jmlr.org/papers/v25/23-0446.html" rel="nofollow noreferrer">https://www.jmlr.org/papers/v25/23-0446.html</a></li>
  <li><a href="https://arxiv.org/abs/2303.12931" rel="nofollow noreferrer">https://arxiv.org/abs/2303.12931</a></li>
  </ul>
  <p>In the Data Thinning paper they reference other research relating to single cell sequencing, and it looks like that has now been published:</p>
  <ul>
  <li><a href="https://doi.org/10.1093/biostatistics/kxac047" rel="nofollow noreferrer">https://doi.org/10.1093/biostatistics/kxac047</a></li>
  </ul>
  <blockquote>
  <p>Under a Poisson assumption, count splitting provides a flexible framework for carrying out valid inference after latent variable estimation that can be applied to virtually any latent variable estimation method and inference technique. This has important applications in the growing field of pseudotime or trajectory analysis, as well as in cell type analysis.</p>
  </blockquote>
  <blockquote>
  <p>We expect count splitting to be useful in situations other than those considered in this article. For example, as explored in Appendix C of the Supplementary material available at Biostatistics online,
  count splitting can be used to test the overall difference in means between two estimated clusters, as considered by Gao and others (2022). As suggested in related work by Batson and others (2019), Chen
  and others (2021), and Gerard (2020), count splitting could be used for model selection tasks such as choosing how many dimensions to keep when reducing the dimension of Poisson scRNA-seq data.</p>
  </blockquote>
  <blockquote>
  <p>Finally, count splitting may lead to power improvements over sample splitting for Poisson data on tasks where the latter is an option (Leiner and others, 2022)</p>
  </blockquote>
  <p>[the <em>latent variable</em> mentioned here can be a qualitative one, such as &quot;cell cluster&quot;]</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-12T21:27:06.023Z
Answers.LastActivityDate: 2024-07-15T21:45:51.417Z
Answers.LastEditDate: 2024-07-15T21:45:51.417Z
Answers.UserDisplayName: gringer
Body: <p><em>This question was also asked on <a href="https://www.reddit.com/r/bioinformatics/comments/1e0glzq/how_can_i_quantitatively_evaluate_which_umap_is/" rel="nofollow noreferrer">Reddit</a></em></p>
  <p>I am new to sc-rna analysis, I have the dataset that I am trying to find out the best UMAP, experimented on trying out different values of the parameters as in (<code>n_neighbors_range = [15, 20, 50, 100, 200]</code>, <code>min_dist_range = [0.1, 0.25, 0.5, 0.8, 0.99]</code>, <code>resolution_range = [0.4, 0.6, 0.8, 1.0, 1.4]</code>) and just to note that the dimensionality reduction done with PCA and visualizing it with UMAP (at least this my understanding). My question is that, how can I evaluate the quality of the embeddings and clustering quantitatively to find out which UMAP is the best? Is that a regular process that you do, and follow? I found out about applying Silhouette score, Calinski Harabasz Index, Davies Bouldin Index, Neighborhood Preservation, Trustworthiness? I'm naive to the data analysis, so I greatly appreciate a feedback and recommendations.</p>
  <p>Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-12T21:30:49.040Z
Comments.Text: Please clarify this question. You're asking about UMAP (a visualisation technique) in the title, but mention resolution range in the parameters, which is a parameter that I am only used to seeing for clustering. Providing additional context around your dataset is always helpful for resolving ambiguities in the question, especially if you can provide a code example showing what you have tried.
Comments.UserDisplayName: gringer
Tags: <single-cell><clustering><pca><umap><data>
Title: How can I quantitatively evaluate which UMAP is best in terms of clustering & embedding?
ThreadText: # How can I quantitatively evaluate which UMAP is best in terms of clustering & embedding?
  
  (Asked by: username_0 on 2024-07-11)
  
  *This question was also asked on [Reddit](https://www.reddit.com/r/bioinformatics/comments/1e0glzq/how_can_i_quantitatively_evaluate_which_umap_is/)*
  
  I am new to sc-rna analysis, I have the dataset that I am trying to find out the best UMAP, experimented on trying out different values of the parameters as in (`n_neighbors_range = [15, 20, 50, 100, 200]`, `min_dist_range = [0.1, 0.25, 0.5, 0.8, 0.99]`, `resolution_range = [0.4, 0.6, 0.8, 1.0, 1.4]`) and just to note that the dimensionality reduction done with PCA and visualizing it with UMAP (at least this my understanding). My question is that, how can I evaluate the quality of the embeddings and clustering quantitatively to find out which UMAP is the best? Is that a regular process that you do, and follow? I found out about applying Silhouette score, Calinski Harabasz Index, Davies Bouldin Index, Neighborhood Preservation, Trustworthiness? I'm naive to the data analysis, so I greatly appreciate a feedback and recommendations.
  
  Thanks!
  
  **Tags:** single-cell, clustering, pca, umap, data
  
  > **Comments:**
  > **username_1**: Please clarify this question. You're asking about UMAP (a visualisation technique) in the title, but mention resolution range in the parameters, which is a parameter that I am only used to seeing for clustering. Providing additional context around your dataset is always helpful for resolving ambiguities in the question, especially if you can provide a code example showing what you have tried.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-12. Score: 1
  
  *[pasting my Reddit answer here]*
  
  You don't. UMAP is a data visualisation technique, not a clustering technique. UMAP has little function beyond providing a way to plot thousands of multi-dimensional data points in an aesthetically-pleasing way.
  
  What you're asking is like asking how to quantitatively evaluate the layout of a histogram.
  
  ---
  
  If, on the other hand, you're asking about *clustering* rather than *visualisation*, then Daniela Witten's group has developed something they call Data Thinning, which may be of use to you:
  
  * <https://www.jmlr.org/papers/v25/23-0446.html>
  * <https://arxiv.org/abs/2303.12931>
  
  In the Data Thinning paper they reference other research relating to single cell sequencing, and it looks like that has now been published:
  
  * <https://doi.org/10.1093/biostatistics/kxac047>
  
  > Under a Poisson assumption, count splitting provides a flexible framework for carrying out valid inference after latent variable estimation that can be applied to virtually any latent variable estimation method and inference technique. This has important applications in the growing field of pseudotime or trajectory analysis, as well as in cell type analysis.
  
  > We expect count splitting to be useful in situations other than those considered in this article. For example, as explored in Appendix C of the Supplementary material available at Biostatistics online,
  > count splitting can be used to test the overall difference in means between two estimated clusters, as considered by Gao and others (2022). As suggested in related work by Batson and others (2019), Chen
  > and others (2021), and Gerard (2020), count splitting could be used for model selection tasks such as choosing how many dimensions to keep when reducing the dimension of Poisson scRNA-seq data.
  
  > Finally, count splitting may lead to power improvements over sample splitting for Poisson data on tasks where the latter is an option (Leiner and others, 2022)
  
  [the *latent variable* mentioned here can be a qualitative one, such as "cell cluster"]
  
  
  ---
  
  