Answers.Body: <p>Scarf has been developed to specifically solve this issue. Scarf handles  large-scale scRNA-Seq/scATAC-Seq/CITE-Seq datasets that will otherwise not fit into memory. Unfortunately, this is not available in R yet.</p>
  <p>Here is the Github link:
  <a href="https://github.com/parashardhapola/scarf" rel="nofollow noreferrer">https://github.com/parashardhapola/scarf</a></p>
  <p>Documentation:
  <a href="https://scarf.readthedocs.io/" rel="nofollow noreferrer">https://scarf.readthedocs.io/</a></p>
  <p>Paper:
  <a href="https://www.nature.com/articles/s41467-022-32097-3" rel="nofollow noreferrer">https://www.nature.com/articles/s41467-022-32097-3</a></p>
  <p>Please feel free to create an issue on Github for you use case (if you cant already find a solution in the documentation).</p>
  <p>/PD</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-15T16:45:11.827Z
Answers.LastActivityDate: 2022-08-15T16:45:41.150Z
Answers.LastEditDate: 2022-08-15T16:45:41.150Z
Answers.UserDisplayName: Parashar
Body: <p>I have several hundred scRNA-seq count matrices, each from a different sample. For my other dataset containg a few dozen samples, I simply merged everything together into one Seurat object, but that won’t work here as far as I can tell. When I try to merge them in the typical way, I get the error:</p>
  <pre><code>Error in .cbind2Csp(x, y) : 
    Cholmod error 'problem too large' at file ../Core/cholmod_sparse.c, line 89
  </code></pre>
  <p>This basically means I ran out of RAM.  I even ran it on the cluster with 1 TB of RAM, and I got the same error.  This is unsurprising, as memory requirements increase exponentially in relation to matrix size.  It seems that this happens when the <em>merge()</em> function attempts to convert all the sparse matrices to dense matrices.  Here is my code:</p>
  <pre class="lang-r prettyprint-override"><code>seurat_object_list &lt;- &quot;path/to/matrices&quot; |&gt; list.dirs(recursive = FALSE) |&gt; lapply(\(matrix_folder) {
      matrix_folder |&gt; Seurat::Read10X() |&gt; Seurat::CreateSeuratObject()
  })
  seurat &lt;- merge(seurat_object_list[[1]], seurat_object_list[2 : length(seurat_object_list)])
  </code></pre>
  <p>The Seurat object list is created with no issue, but it always fails on the merge.  I tested this code with a smaller subset of the matrices, and it works just fine.  It seems that I just have too many cells to process in this manner.</p>
  <p>At this point, It looks like I have a few options.</p>
  <ol>
  <li>I could hack together some way to force the matrices into one Seurat object.  But I don't know if that would even be useful, since I would likely keep bumping my head on the same memory contraints whenever I try to do anything with the combined object, such as running <em>SCTransform()</em> or <em>FindMarkers()</em>.</li>
  <li>I could divide the samples into groups that are small enough to merge, and process them in the normal way.  But this imposes limitations on the types of comparisons/analyses I could perform.</li>
  <li>I could keep the matrices separate and analyze them iteratively.  This would solve the memory problems, but I haven't found any info online about anyone analyzing count matrices this way.  I imagine that I would lose a lot of out-of-the-box functionality that comes with the standard single-cell R libraries.  For example, I'm not sure how I would normalize all the gene counts against each other as separate objects.  This option seems like it might require a lot of re-inventing the wheel and a lot of code.</li>
  </ol>
  <p>I like option 3 the best, but I'm not sure where to begin.  I am still pretty new to bioninformatics, so I would love to hear some input from those who are more experienced.  How do you deal with memory constraints when dealing with a large number of cells/samples?  Are there any standard ways to deal with this problem?  Is there some tutorial/doc online that I failed to find?  Any advice you could offer is greatly appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-02T10:22:57.297Z
Comments.Text: I get the same error but not running out of memory. I don't think this is a memory problem. It's some internal limitation of dgCMatrix object.
Comments.UserDisplayName: mindlessgreen
Tags: <r><scrnaseq><seurat>
Title: How to manage memory contraints when analyzing a large number of gene count matrices? I keep running out of RAM with my current pipeline
ThreadText: # How to manage memory contraints when analyzing a large number of gene count matrices? I keep running out of RAM with my current pipeline
  
  (Asked by: username_0 on 2022-05-02)
  
  I have several hundred scRNA-seq count matrices, each from a different sample. For my other dataset containg a few dozen samples, I simply merged everything together into one Seurat object, but that won’t work here as far as I can tell. When I try to merge them in the typical way, I get the error:
  
  ```
  Error in .cbind2Csp(x, y) : 
    Cholmod error 'problem too large' at file ../Core/cholmod_sparse.c, line 89
  
  ```
  
  This basically means I ran out of RAM. I even ran it on the cluster with 1 TB of RAM, and I got the same error. This is unsurprising, as memory requirements increase exponentially in relation to matrix size. It seems that this happens when the *merge()* function attempts to convert all the sparse matrices to dense matrices. Here is my code:
  
  ```r
  seurat_object_list <- "path/to/matrices" |> list.dirs(recursive = FALSE) |> lapply(\(matrix_folder) {
      matrix_folder |> Seurat::Read10X() |> Seurat::CreateSeuratObject()
  })
  seurat <- merge(seurat_object_list[[1]], seurat_object_list[2 : length(seurat_object_list)])
  
  ```
  
  The Seurat object list is created with no issue, but it always fails on the merge. I tested this code with a smaller subset of the matrices, and it works just fine. It seems that I just have too many cells to process in this manner.
  
  At this point, It looks like I have a few options.
  
  1. I could hack together some way to force the matrices into one Seurat object. But I don't know if that would even be useful, since I would likely keep bumping my head on the same memory contraints whenever I try to do anything with the combined object, such as running *SCTransform()* or *FindMarkers()*.
  2. I could divide the samples into groups that are small enough to merge, and process them in the normal way. But this imposes limitations on the types of comparisons/analyses I could perform.
  3. I could keep the matrices separate and analyze them iteratively. This would solve the memory problems, but I haven't found any info online about anyone analyzing count matrices this way. I imagine that I would lose a lot of out-of-the-box functionality that comes with the standard single-cell R libraries. For example, I'm not sure how I would normalize all the gene counts against each other as separate objects. This option seems like it might require a lot of re-inventing the wheel and a lot of code.
  
  I like option 3 the best, but I'm not sure where to begin. I am still pretty new to bioninformatics, so I would love to hear some input from those who are more experienced. How do you deal with memory constraints when dealing with a large number of cells/samples? Are there any standard ways to deal with this problem? Is there some tutorial/doc online that I failed to find? Any advice you could offer is greatly appreciated.
  
  **Tags:** r, scrnaseq, seurat
  
  > **Comments:**
  > **username_1**: I get the same error but not running out of memory. I don't think this is a memory problem. It's some internal limitation of dgCMatrix object.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-08-15. Score: 1
  
  Scarf has been developed to specifically solve this issue. Scarf handles large-scale scRNA-Seq/scATAC-Seq/CITE-Seq datasets that will otherwise not fit into memory. Unfortunately, this is not available in R yet.
  
  Here is the Github link:
  <https://github.com/parashardhapola/scarf>
  
  Documentation:
  <https://scarf.readthedocs.io/>
  
  Paper:
  <https://www.nature.com/articles/s41467-022-32097-3>
  
  Please feel free to create an issue on Github for you use case (if you cant already find a solution in the documentation).
  
  /PD
  
  
  ---
  
  