Answers.Body: <p>You could try the PBMC 3k data from the Satija lab:</p>
  <p><a href="https://satijalab.org/seurat/articles/pbmc3k_tutorial.html" rel="nofollow noreferrer">https://satijalab.org/seurat/articles/pbmc3k_tutorial.html</a></p>
  <p>For well-annotated data, there's the Single-cell proteo-genomic reference map:</p>
  <p><a href="https://www.biorxiv.org/content/10.1101/2021.03.18.435922v1" rel="nofollow noreferrer">https://www.biorxiv.org/content/10.1101/2021.03.18.435922v1</a></p>
  <p>Data for that are available <a href="https://figshare.com/projects/Single-cell_proteo-genomic_reference_maps_of_the_human_hematopoietic_system/94469" rel="nofollow noreferrer">here</a>.</p>
  <p>If these datasets are too large (i.e. too good), you can use the existing counts as weighting for a random removal of reads. Genes with high expression are likely to be present regardless of sequencing depth, hence the count-adjusted removal.</p>
  <p>FWIW, I don't think that supervised learning is a good fit for dealing with data sparsity in cDNA read counts. There are plenty of existing attempts to solve the issues by adjusting the assumed models, including research by the authors of commonly-used bulk differential expression tools (e.g. <a href="https://doi.org/10.1186/s13059-018-1406-4" rel="nofollow noreferrer">here</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-25T11:07:19.290Z
Answers.Text: Thanks! I like the second reference, it is fresh from the oven. If the data is "too good" (i.e. traditional models can easily distinguish the different cell types), you would then introduce random dropouts to increase the sparsity?
Answers.UserDisplayName: rvinas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-25T20:57:01.243Z
Answers.Text: Random dropouts at the *read* level; I wouldn't randomly drop out genes.
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-25T10:28:35.267Z
Answers.LastActivityDate: 2021-03-25T20:58:10.183Z
Answers.LastEditDate: 2021-03-25T20:58:10.183Z
Answers.UserDisplayName: gringer
Body: <p>One challenging aspect of modeling scRNA-seq data is data sparsity, that is, scRNA-seq measurements typically suffer from large fractions of observed zeros (i.e. <em>dropouts</em>), where a given gene in a given cell has no unique molecular identifiers or reads mapping to it [<a href="https://genomebiology.biomedcentral.com/articles/10.1186/s13059-020-1926-6" rel="nofollow noreferrer">1</a>].</p>
  <p>What would be an appropriate dataset to benchmark supervised learning algorithms that operate on single-cell RNA-seq data? In particular, I am looking for challenging datasets (preferably with labels, i.e. cell type classification) where existing methods struggle due to the sparsity of the data. Additionally, does it ever happen that the <em>dropouts</em> are confounders of the labels? (i.e. for a given cell type A the <em>dropout</em> probability of a gene X is higher than for another cell type B).</p>
  <p>Any pointers are welcome.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-24T22:11:11.497Z
Comments.Text: Can you say what is the purpose of your supervised learning method? to annotate cell types?
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-24T22:40:47.610Z
Comments.Text: Yes, that's right, cell type annotation is the first thing that comes to mind, but I am also willing to consider other tasks (e.g. classification of cancer cells). Even regression tasks would be fine too. Essentially, I have a method in mind that might be useful for modeling non-linear+sparse data and I am looking for an appropriate benchmark dataset.
Comments.UserDisplayName: rvinas
Tags: <rna-seq><scrnaseq><single-cell><machine-learning><benchmarking>
Title: Challenging benchmarks for supervised learning on sparse scRNA-seq data
ThreadText: # Challenging benchmarks for supervised learning on sparse scRNA-seq data
  
  (Asked by: username_0 on 2021-03-24)
  
  One challenging aspect of modeling scRNA-seq data is data sparsity, that is, scRNA-seq measurements typically suffer from large fractions of observed zeros (i.e. *dropouts*), where a given gene in a given cell has no unique molecular identifiers or reads mapping to it [[1](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-020-1926-6)].
  
  What would be an appropriate dataset to benchmark supervised learning algorithms that operate on single-cell RNA-seq data? In particular, I am looking for challenging datasets (preferably with labels, i.e. cell type classification) where existing methods struggle due to the sparsity of the data. Additionally, does it ever happen that the *dropouts* are confounders of the labels? (i.e. for a given cell type A the *dropout* probability of a gene X is higher than for another cell type B).
  
  Any pointers are welcome.
  
  **Tags:** rna-seq, scrnaseq, single-cell, machine-learning, benchmarking
  
  > **Comments:**
  > **username_1**: Can you say what is the purpose of your supervised learning method? to annotate cell types?  
  > **username_0**: Yes, that's right, cell type annotation is the first thing that comes to mind, but I am also willing to consider other tasks (e.g. classification of cancer cells). Even regression tasks would be fine too. Essentially, I have a method in mind that might be useful for modeling non-linear+sparse data and I am looking for an appropriate benchmark dataset.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-25. Score: 2 | Accepted answer
  
  You could try the PBMC 3k data from the Satija lab:
  
  <https://satijalab.org/seurat/articles/pbmc3k_tutorial.html>
  
  For well-annotated data, there's the Single-cell proteo-genomic reference map:
  
  <https://www.biorxiv.org/content/10.1101/2021.03.18.435922v1>
  
  Data for that are available [here](https://figshare.com/projects/Single-cell_proteo-genomic_reference_maps_of_the_human_hematopoietic_system/94469).
  
  If these datasets are too large (i.e. too good), you can use the existing counts as weighting for a random removal of reads. Genes with high expression are likely to be present regardless of sequencing depth, hence the count-adjusted removal.
  
  FWIW, I don't think that supervised learning is a good fit for dealing with data sparsity in cDNA read counts. There are plenty of existing attempts to solve the issues by adjusting the assumed models, including research by the authors of commonly-used bulk differential expression tools (e.g. [here](https://doi.org/10.1186/s13059-018-1406-4)).
  
  > **Comments:**
  > **username_0**: Thanks! I like the second reference, it is fresh from the oven. If the data is "too good" (i.e. traditional models can easily distinguish the different cell types), you would then introduce random dropouts to increase the sparsity?  
  > **username_2**: Random dropouts at the *read* level; I wouldn't randomly drop out genes.  
  
  ---
  
  