Answers.Body: <p>Not a direct solution but some workarounds:</p>
  <ul>
  <li><p>As far as I know, <code>Seurat</code> can work with sparse matrices.</p>
  </li>
  <li><p>The particular function of <code>scran</code> that you are using eats up quite some memory. I believe it is needed for the &quot;normalization&quot; step (that is how I used it anyway). Whereas the scaling normalization performed by this function is superior to crude &quot;log normalization&quot;, you can give a try with the latter, which is far less computationally-intensive (does not do clustering). Seurat, once again, can help with this.</p>
  </li>
  <li><p>You can downsample your data to the extent that it fits your into your RAM.</p>
  </li>
  <li><p>You can give a try with Python. More and more single cell packages are written in Python, to some extent because of the problem you have experienced. For example <code>Scanpy</code> output is comparable to that of <code>Seurat</code>, although I am not sure if you can use scaling normalization with <code>Scanpy</code>.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T15:11:16.710Z
Answers.Text: I solved the issue. Anyway I will try what you suggests also because it is instructive. Thank you very much!
Answers.UserDisplayName: Spartan 117
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T15:19:58.917Z
Answers.Text: Please let the community know how have you solved the issue, this is important for people who will encounter the same problem. You can just "answer" your own question and "accept" the answer.
Answers.UserDisplayName: haci
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T11:46:51.167Z
Answers.LastActivityDate: 2020-07-06T13:53:31.817Z
Answers.LastEditDate: 2020-07-06T13:53:31.817Z
Answers.UserDisplayName: haci
Answers.Body: <p>Essentially you've hit a RAM bottleneck and the calculation will slow to zero, or in this instance refuse to go forward. The way to do this normally is to parallelize the calculation across the cores of your machine. This will likely remove the RAM bottlenack, dont ask me the computer archetectural reasons for why it works - but its works.</p>
  <p>However, my knowledge of R is minimal. I wouldn't know how to parallelize an R calculation. It is certainly doable in Perl and Python, but the calculation needs to be written to ensure parallelisation.</p>
  <p>The other way in is to reconfigure your calculation to remove sparse matrices OR find someone doing NGS where they've confgured their machine around heavy RAM.</p>
  <p>Looking at your calculation I don't quite understand why you need to use a specific package, it looks like unsupervised machine learning, PCA - tSNE that sort of thing and you don't need a given package to do that you just need to vectorise the inputs. If you worked out the statistical components of <code>Scran</code> then there are a few extremely strong R statisticians/bioinformaticians on site who would have zero problem in replicating this within a few lines of code. Its not hard in Python's Sci-kit learn either. At a guess they perform PCA and resolve it via tSNE and this give nice clear clusters.</p>
  <p>GCP is free for 3 months so it will cost you zero in context to a single calculation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T15:10:14.807Z
Answers.Text: Even PCA gives problem if it ingests this sce-object. Anyway I solved changing the computer with one more performative. Very good to know that GCP is free for three months. Thank you!
Answers.UserDisplayName: Spartan 117
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T16:10:47.263Z
Answers.Text: I prefer @haci 's work around, thanks are not a cool way to say thanks.but anyway
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T17:49:40.807Z
Answers.Text: good suggestions @Michael. It's not a big dataset, 3500+ cells, so you can do PCA or tSNE on the dimension reduced data. I don't see the OP's calculation though, or what is the larger question..
Answers.UserDisplayName: StupidWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T19:14:23.377Z
Answers.Text: .. me neither but they are doing Plasmodium (malaria, P. falciparum?) so there will be 24-hour periodicity in the RNA expression depending on what in vitro assay they are doing. That would make it a bit complicated. They also appear to do martial arts - its a tough field malaria ;-)
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T10:58:58.817Z
Answers.LastActivityDate: 2020-07-06T11:35:32.177Z
Answers.LastEditDate: 2020-07-06T11:35:32.177Z
Answers.UserDisplayName: M__
Answers.Body: <p>Not a direct solution to your problem because it does not use SCE. But if you (and others who ended on this post) are facing the RAM bottleneck when analysing single-cell data, then <strong>Scarf</strong> might be useful.</p>
  <p>Here is the Github link:
  <a href="https://github.com/parashardhapola/scarf" rel="nofollow noreferrer">https://github.com/parashardhapola/scarf</a></p>
  <p>Documentation:
  <a href="https://scarf.readthedocs.io/" rel="nofollow noreferrer">https://scarf.readthedocs.io/</a></p>
  <p>Paper:
  <a href="https://www.nature.com/articles/s41467-022-32097-3" rel="nofollow noreferrer">https://www.nature.com/articles/s41467-022-32097-3</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-15T16:36:48.157Z
Answers.LastActivityDate: 2022-08-15T16:36:48.157Z
Answers.UserDisplayName: Parashar
Answers.Body: <p>Ah, looks like I can't even procrastinate on StackExchange anymore without seeing work-related stuff. Oh well.</p>
  <p>Anyway, the other answers and comments are way off. <strong>scran</strong> has supported sparse matrices for years, ever since we switched over to the <code>SingleCellExperiment</code> class as our basic data structure. <code>quickCluster</code> does no coercion to dense format unless you tell it so explicitly, e.g., with <code>use.ranks=TRUE</code> (in which case you're asking for ranks, so there's little choice <em>but</em> to collapse to a dense matrix).</p>
  <p>You don't provide an MWE or your session information, but this is how it rolls for me:</p>
  <pre class="lang-r prettyprint-override"><code># Using the raw counts in the linked dataset. Despite being
  # called a CSV, it's actually space delimited... typical.
  library(scater)
  mat &lt;- readSparseCounts(&quot;GBM_raw_gene_counts.csv&quot;, sep=&quot; &quot;)
  
  # Making an SCE just for fun. Not strictly necessary for
  # this example, but you'll find it useful later. 
  sce &lt;- SingleCellExperiment(list(counts=mat))
  
  library(scran)
  system.time(clust &lt;- quickCluster(sce))
  ##   user  system elapsed
  ##  3.170   0.174   3.411
  </code></pre>
  <p>This is running on my laptop - 16 GB RAM but I'm definitely not using all of it. I only go full throttle when I'm working on some real data, e.g., the 300k HCA bone marrow dataset. Check out the <a href="https://osca.bioconductor.org/" rel="noreferrer">book</a> for more details.</p>
  <p>Session info below, I don't know quite enough SO-fu to know collapse it.</p>
  <pre><code>R version 4.0.0 Patched (2020-04-27 r78316)
  Platform: x86_64-apple-darwin17.7.0 (64-bit)
  Running under: macOS High Sierra 10.13.6
  
  Matrix products: default
  BLAS:   /Users/luna/Software/R/R-4-0-branch/lib/libRblas.dylib
  LAPACK: /Users/luna/Software/R/R-4-0-branch/lib/libRlapack.dylib
  
  locale:
  [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
  
  attached base packages:
  [1] parallel  stats4    stats     graphics  grDevices utils     datasets
  [8] methods   base
  
  other attached packages:
   [1] scran_1.16.0                scater_1.16.1
   [3] ggplot2_3.3.2               SingleCellExperiment_1.10.1
   [5] SummarizedExperiment_1.18.1 DelayedArray_0.14.0
   [7] matrixStats_0.56.0          Biobase_2.48.0
   [9] GenomicRanges_1.40.0        GenomeInfoDb_1.24.2
  [11] IRanges_2.22.2              S4Vectors_0.26.1
  [13] BiocGenerics_0.34.0
  
  loaded via a namespace (and not attached):
   [1] beeswarm_0.2.3            statmod_1.4.34
   [3] tidyselect_1.1.0          locfit_1.5-9.4
   [5] purrr_0.3.4               BiocSingular_1.4.0
   [7] lattice_0.20-41           colorspace_1.4-1
   [9] vctrs_0.3.1               generics_0.0.2
  [11] viridisLite_0.3.0         rlang_0.4.6
  [13] pillar_1.4.4              glue_1.4.1
  [15] withr_2.2.0               BiocParallel_1.22.0
  [17] dqrng_0.2.1               GenomeInfoDbData_1.2.3
  [19] lifecycle_0.2.0           zlibbioc_1.34.0
  [21] munsell_0.5.0             gtable_0.3.0
  [23] rsvd_1.0.3                vipor_0.4.5
  [25] irlba_2.3.3               BiocNeighbors_1.6.0
  [27] Rcpp_1.0.4.6              edgeR_3.30.3
  [29] scales_1.1.1              limma_3.44.3
  [31] XVector_0.28.0            gridExtra_2.3
  [33] dplyr_1.0.0               grid_4.0.0
  [35] tools_4.0.0               bitops_1.0-6
  [37] magrittr_1.5              RCurl_1.98-1.2
  [39] tibble_3.0.1              crayon_1.3.4
  [41] pkgconfig_2.0.3           ellipsis_0.3.1
  [43] Matrix_1.2-18             DelayedMatrixStats_1.10.0
  [45] ggbeeswarm_0.6.0          viridis_0.5.1
  [47] R6_2.4.1                  igraph_1.2.5
  [49] compiler_4.0.0
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T21:08:22.150Z
Answers.LastActivityDate: 2020-07-06T21:08:22.150Z
Answers.UserDisplayName: wizard_of_oz
Body: <p>I'm dealing with a very large and sparse dataset and the first issues I met occurred when I tried to use quickCluster that reported me this error:</p>
  <pre><code>                'cannot allocate vector of size 156.6 Mb'
  </code></pre>
  <p>So, given that I cannot wait to change the RAM of my computer and I can't afford to use a cluster, I want to rely on some other strategies like some package that would allow me to handle sparse matrices. I'm thinking about sparseM but given that I don't know well this package I'd like to know how to shrink the ram allocation for these kind of matrices. Any suggestion will be very appreciated!</p>
  <p><a href="http://www.gbmseq.org" rel="nofollow noreferrer">Link to the dataset</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-06T10:36:23.440Z
Comments.Text: This is with the scran package, yes? Can you update your tags to mention this? To my knowledge scran doesn't support sparse matrix input, which will be a bit of a show-stopper unless you can get access to a system with more RAM (maybe you can pay a few bucks for an AWS instance?).
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-06T10:44:45.907Z
Comments.Text: Yes it is. So you are telling me to utilize AWS?This could be a problem since it is a thesis and I do not want to pay.
Comments.UserDisplayName: Spartan 117
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-06T10:45:42.073Z
Comments.Text: You could go to GCP which is cheaper than AWS (Google Cloud). How many cores do you have ?
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-06T10:56:09.813Z
Comments.Text: 4 cores I presume
Comments.UserDisplayName: Spartan 117
Tags: <rna-seq><scrnaseq><bioconductor><single-cell><scran>
Title: How to reduce the occupied RAM when you are dealing with a very sparse matrix in a single-cell Experiment in R?
ThreadText: # How to reduce the occupied RAM when you are dealing with a very sparse matrix in a single-cell Experiment in R?
  
  (Asked by: username_0 on 2020-07-06)
  
  I'm dealing with a very large and sparse dataset and the first issues I met occurred when I tried to use quickCluster that reported me this error:
  
  ```
                  'cannot allocate vector of size 156.6 Mb'
  
  ```
  
  So, given that I cannot wait to change the RAM of my computer and I can't afford to use a cluster, I want to rely on some other strategies like some package that would allow me to handle sparse matrices. I'm thinking about sparseM but given that I don't know well this package I'd like to know how to shrink the ram allocation for these kind of matrices. Any suggestion will be very appreciated!
  
  [Link to the dataset](http://www.gbmseq.org)
  
  **Tags:** rna-seq, scrnaseq, bioconductor, single-cell, scran
  
  > **Comments:**
  > **username_1**: This is with the scran package, yes? Can you update your tags to mention this? To my knowledge scran doesn't support sparse matrix input, which will be a bit of a show-stopper unless you can get access to a system with more RAM (maybe you can pay a few bucks for an AWS instance?).  
  > **username_0**: Yes it is. So you are telling me to utilize AWS?This could be a problem since it is a thesis and I do not want to pay.  
  > **username_2**: You could go to GCP which is cheaper than AWS (Google Cloud). How many cores do you have ?  
  > **username_0**: 4 cores I presume  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_6 on 2020-07-06. Score: 5 | Accepted answer
  
  Ah, looks like I can't even procrastinate on StackExchange anymore without seeing work-related stuff. Oh well.
  
  Anyway, the other answers and comments are way off. **scran** has supported sparse matrices for years, ever since we switched over to the `SingleCellExperiment` class as our basic data structure. `quickCluster` does no coercion to dense format unless you tell it so explicitly, e.g., with `use.ranks=TRUE` (in which case you're asking for ranks, so there's little choice *but* to collapse to a dense matrix).
  
  You don't provide an MWE or your session information, but this is how it rolls for me:
  
  ```r
  # Using the raw counts in the linked dataset. Despite being
  # called a CSV, it's actually space delimited... typical.
  library(scater)
  mat <- readSparseCounts("GBM_raw_gene_counts.csv", sep=" ")
  
  # Making an SCE just for fun. Not strictly necessary for
  # this example, but you'll find it useful later. 
  sce <- SingleCellExperiment(list(counts=mat))
  
  library(scran)
  system.time(clust <- quickCluster(sce))
  ##   user  system elapsed
  ##  3.170   0.174   3.411
  
  ```
  
  This is running on my laptop - 16 GB RAM but I'm definitely not using all of it. I only go full throttle when I'm working on some real data, e.g., the 300k HCA bone marrow dataset. Check out the [book](https://osca.bioconductor.org/) for more details.
  
  Session info below, I don't know quite enough SO-fu to know collapse it.
  
  ```
  R version 4.0.0 Patched (2020-04-27 r78316)
  Platform: x86_64-apple-darwin17.7.0 (64-bit)
  Running under: macOS High Sierra 10.13.6
  
  Matrix products: default
  BLAS:   /Users/luna/Software/R/R-4-0-branch/lib/libRblas.dylib
  LAPACK: /Users/luna/Software/R/R-4-0-branch/lib/libRlapack.dylib
  
  locale:
  [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
  
  attached base packages:
  [1] parallel  stats4    stats     graphics  grDevices utils     datasets
  [8] methods   base
  
  other attached packages:
   [1] scran_1.16.0                scater_1.16.1
   [3] ggplot2_3.3.2               SingleCellExperiment_1.10.1
   [5] SummarizedExperiment_1.18.1 DelayedArray_0.14.0
   [7] matrixStats_0.56.0          Biobase_2.48.0
   [9] GenomicRanges_1.40.0        GenomeInfoDb_1.24.2
  [11] IRanges_2.22.2              S4Vectors_0.26.1
  [13] BiocGenerics_0.34.0
  
  loaded via a namespace (and not attached):
   [1] beeswarm_0.2.3            statmod_1.4.34
   [3] tidyselect_1.1.0          locfit_1.5-9.4
   [5] purrr_0.3.4               BiocSingular_1.4.0
   [7] lattice_0.20-41           colorspace_1.4-1
   [9] vctrs_0.3.1               generics_0.0.2
  [11] viridisLite_0.3.0         rlang_0.4.6
  [13] pillar_1.4.4              glue_1.4.1
  [15] withr_2.2.0               BiocParallel_1.22.0
  [17] dqrng_0.2.1               GenomeInfoDbData_1.2.3
  [19] lifecycle_0.2.0           zlibbioc_1.34.0
  [21] munsell_0.5.0             gtable_0.3.0
  [23] rsvd_1.0.3                vipor_0.4.5
  [25] irlba_2.3.3               BiocNeighbors_1.6.0
  [27] Rcpp_1.0.4.6              edgeR_3.30.3
  [29] scales_1.1.1              limma_3.44.3
  [31] XVector_0.28.0            gridExtra_2.3
  [33] dplyr_1.0.0               grid_4.0.0
  [35] tools_4.0.0               bitops_1.0-6
  [37] magrittr_1.5              RCurl_1.98-1.2
  [39] tibble_3.0.1              crayon_1.3.4
  [41] pkgconfig_2.0.3           ellipsis_0.3.1
  [43] Matrix_1.2-18             DelayedMatrixStats_1.10.0
  [45] ggbeeswarm_0.6.0          viridis_0.5.1
  [47] R6_2.4.1                  igraph_1.2.5
  [49] compiler_4.0.0
  
  ```
  
  
  ---
  
  ### Answer by username_3 on 2020-07-06. Score: 3
  
  Not a direct solution but some workarounds:
  
  * As far as I know, `Seurat` can work with sparse matrices.
  * The particular function of `scran` that you are using eats up quite some memory. I believe it is needed for the "normalization" step (that is how I used it anyway). Whereas the scaling normalization performed by this function is superior to crude "log normalization", you can give a try with the latter, which is far less computationally-intensive (does not do clustering). Seurat, once again, can help with this.
  * You can downsample your data to the extent that it fits your into your RAM.
  * You can give a try with Python. More and more single cell packages are written in Python, to some extent because of the problem you have experienced. For example `Scanpy` output is comparable to that of `Seurat`, although I am not sure if you can use scaling normalization with `Scanpy`.
  
  > **Comments:**
  > **username_0**: I solved the issue. Anyway I will try what you suggests also because it is instructive. Thank you very much!  
  > **username_3**: Please let the community know how have you solved the issue, this is important for people who will encounter the same problem. You can just "answer" your own question and "accept" the answer.  
  
  ---
  
  ### Answer by username_2 on 2020-07-06. Score: 1
  
  Essentially you've hit a RAM bottleneck and the calculation will slow to zero, or in this instance refuse to go forward. The way to do this normally is to parallelize the calculation across the cores of your machine. This will likely remove the RAM bottlenack, dont ask me the computer archetectural reasons for why it works - but its works.
  
  However, my knowledge of R is minimal. I wouldn't know how to parallelize an R calculation. It is certainly doable in Perl and Python, but the calculation needs to be written to ensure parallelisation.
  
  The other way in is to reconfigure your calculation to remove sparse matrices OR find someone doing NGS where they've confgured their machine around heavy RAM.
  
  Looking at your calculation I don't quite understand why you need to use a specific package, it looks like unsupervised machine learning, PCA - tSNE that sort of thing and you don't need a given package to do that you just need to vectorise the inputs. If you worked out the statistical components of `Scran` then there are a few extremely strong R statisticians/bioinformaticians on site who would have zero problem in replicating this within a few lines of code. Its not hard in Python's Sci-kit learn either. At a guess they perform PCA and resolve it via tSNE and this give nice clear clusters.
  
  GCP is free for 3 months so it will cost you zero in context to a single calculation.
  
  > **Comments:**
  > **username_0**: Even PCA gives problem if it ingests this sce-object. Anyway I solved changing the computer with one more performative. Very good to know that GCP is free for three months. Thank you!  
  > **username_2**: I prefer @username_3 's work around, thanks are not a cool way to say thanks.but anyway  
  > **username_4**: good suggestions @Michael. It's not a big dataset, 3500+ cells, so you can do PCA or tSNE on the dimension reduced data. I don't see the OP's calculation though, or what is the larger question..  
  > **username_2**: .. me neither but they are doing Plasmodium (malaria, P. falciparum?) so there will be 24-hour periodicity in the RNA expression depending on what in vitro assay they are doing. That would make it a bit complicated. They also appear to do martial arts - its a tough field malaria ;-)  
  
  ---
  
  ### Answer by username_5 on 2022-08-15. Score: 0
  
  Not a direct solution to your problem because it does not use SCE. But if you (and others who ended on this post) are facing the RAM bottleneck when analysing single-cell data, then **Scarf** might be useful.
  
  Here is the Github link:
  <https://github.com/parashardhapola/scarf>
  
  Documentation:
  <https://scarf.readthedocs.io/>
  
  Paper:
  <https://www.nature.com/articles/s41467-022-32097-3>
  
  
  ---
  
  