Answers.Body: <p>You can't compute a correlation matrix when you have more than ~46000 rows, since a standard R matrix can have a maximum of 2^31-1 values. Have a look at packages like <a href="https://www.rdocumentation.org/packages/propagate/versions/1.0-6/topics/bigcor" rel="nofollow noreferrer">bigcor</a>. Alternatively, consider if having 3+ billion correlation values is really useful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-06T13:33:02.020Z
Answers.LastActivityDate: 2019-03-06T13:33:02.020Z
Answers.UserDisplayName: Devon Ryan
Body: <p>I am working on a script to create a correlation matrix and mutual_rank matrix from RNAseq gene counts for 57992 genes in 7027 samples. I have already tested this script for a smaller file (for 10,000 genes) and it is working, however, when I am running this script on my final input file, it is giving the following error:</p>
  
  <pre><code> corrupt matrix -- dims not not match length (this error comes after gene to 
   gene correlation step)
  </code></pre>
  
  <p>Script which i am running is:</p>
  
  <pre><code>E &lt;- read.table("mdat_human_test.txt", header = TRUE, row.names=1)
  
  # gene-to-gene correlation
  C &lt;- cor (t(E))
  write.table(C, "human_correlation_genes_mdat.txt", quote=FALSE, sep="\t", 
  row.names=TRUE)
  
  # correlation rank matrix
  R &lt;- t(apply(1-C, 1, rank))
  write.table(R, "human_rank_genes_mdat.txt", quote=FALSE, sep="\t", 
  row.names=TRUE)
  
  # mutual rank matrix
  M &lt;- sqrt(R * t(R))
  write.table(M, "human_mutual_rank_genes_mdat.txt", quote=FALSE, sep="\t", 
  row.names=TRUE)
  
  q()
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-06T13:40:03.403Z
Comments.Text: What is your biological question of interest ? Why are you doing this correlations? In addition to [Devon](https://bioinformatics.stackexchange.com/a/7203/48)'s answer, I would like to point out that if you are using all the human annotated genes you might need to filter and normalize them. You'll presumably will have 25% of "genes" which are poorly described and/or with few counts are you really interested in keeping those ?
Comments.UserDisplayName: llrs
Tags: <r><rna-seq><correlation>
Title: Error in formation of correlation matrix-(corrupt matrix -- dims not not match length)
ThreadText: # Error in formation of correlation matrix-(corrupt matrix -- dims not not match length)
  
  (Asked by: username_0 on 2019-03-06)
  
  I am working on a script to create a correlation matrix and mutual\_rank matrix from RNAseq gene counts for 57992 genes in 7027 samples. I have already tested this script for a smaller file (for 10,000 genes) and it is working, however, when I am running this script on my final input file, it is giving the following error:
  
  ```
   corrupt matrix -- dims not not match length (this error comes after gene to 
   gene correlation step)
  
  ```
  
  Script which i am running is:
  
  ```
  E <- read.table("mdat_human_test.txt", header = TRUE, row.names=1)
  
  # gene-to-gene correlation
  C <- cor (t(E))
  write.table(C, "human_correlation_genes_mdat.txt", quote=FALSE, sep="\t", 
  row.names=TRUE)
  
  # correlation rank matrix
  R <- t(apply(1-C, 1, rank))
  write.table(R, "human_rank_genes_mdat.txt", quote=FALSE, sep="\t", 
  row.names=TRUE)
  
  # mutual rank matrix
  M <- sqrt(R * t(R))
  write.table(M, "human_mutual_rank_genes_mdat.txt", quote=FALSE, sep="\t", 
  row.names=TRUE)
  
  q()
  
  ```
  
  **Tags:** r, rna-seq, correlation
  
  > **Comments:**
  > **username_1**: What is your biological question of interest ? Why are you doing this correlations? In addition to [username_2](https://bioinformatics.stackexchange.com/a/7203/48)'s answer, I would like to point out that if you are using all the human annotated genes you might need to filter and normalize them. You'll presumably will have 25% of "genes" which are poorly described and/or with few counts are you really interested in keeping those ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-03-06. Score: 3
  
  You can't compute a correlation matrix when you have more than ~46000 rows, since a standard R matrix can have a maximum of 2^31-1 values. Have a look at packages like [bigcor](https://www.rdocumentation.org/packages/propagate/versions/1.0-6/topics/bigcor). Alternatively, consider if having 3+ billion correlation values is really useful.
  
  
  ---
  
  