Answers.Body: <p>Your code looks fine, but <code>rbind</code> to progressively build and resize a data frame can be gluttonous in terms of memory and time, especially given the A-row x B-row times you are doing this operation, which is fine for very small datasets, but not often realistic for real-world data.</p>
  
  <p>Instead, perhaps <code>cat</code> results to a text file, where results are written in tab-delimited columns. You can then read in that text file later on. </p>
  
  <p>In addition to serializing or saving your results to text, which make them easier to work with in non-R contexts, you won't have to deal with R's memory allocation problems when running <code>rbind</code> repeatedly. You just write out the answer to a file and read it in later on, in one pass.</p>
  
  <p>Another approach is to preallocate memory to a presized data frame, and then write to that allocated space. </p>
  
  <p>You know the size of your inputs ahead of time, which makes this approach tenable. In this case, it looks like you are storing two floats and a string (<code>rho</code>, <code>p.val</code>, and <code>Gene</code>) and that you are going to store A-row x B-row number of them. </p>
  
  <p>Here's one way you might go about this:</p>
  
  <pre><code># create a presized data frame
  Arows &lt;- nrow(A.data)
  Brows &lt;- nrow(B.data)
  correlation.result &lt;- data.frame(t(matrix(c(0.0, 0.0, "foo"), nrow=3, ncol=Arows * Brows)), stringsAsFactors=FALSE)
  colnames(correlation.result) &lt;- c("rho", "p.val", "Gene")
  
  # assign values to that presized data frame 
  Acols &lt;- ncol(A.data)
  Bcols &lt;- ncol(B.data)
  for (i in 1:Arows) {
    # get A data
    Ai &lt;- as.numeric(A.data[i, 2:Acols])
    Aname &lt;- A.data[i, 1]
    for (j in 1:Brows) {
      # calculate row offset in presized data frame
      ij &lt;- (i - 1) * Brows + j
      # get B data
      Bj &lt;- as.numeric(B.data[j, 2:Bcols])
      Bname &lt;- B.data[j, 1]
      # run test
      correln &lt;- cor.test(Ai, Bj, method = "spearman", exact = FALSE)
      rho &lt;- correln$estimate
      p.val &lt;- correln$p.value
      Gene &lt;- paste(Aname, Bname, sep="_", collapse=" ")
      # assign value to row of result data frame
      correlation.result[ij,] &lt;- c(rho, p.val, Gene)
    }
  }
  </code></pre>
  
  <p>This avoids R needing to resize and recopy the data frame contents as they grow, which I suspect is the major reason it is taking so long to run.</p>
  
  <p>Chapter or "Circle" 2 of Patrick Burns' book <em>The R Inferno</em> discusses the problems with growing objects with <code>rbind</code> and ways to handle it — you might find this a useful read. This book is a PDF download via your search engine of choice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-09T15:24:11.697Z
Answers.Text: Is there a reason for the downvote(s)?
Answers.UserDisplayName: Alex Reynolds
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-11T02:41:43.637Z
Answers.Text: I think it is bad for a community when downvoting without any explanation. Please explain yourself.
Answers.UserDisplayName: Alex Reynolds
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-16T19:48:58.853Z
Answers.LastActivityDate: 2018-06-18T19:49:19.950Z
Answers.LastEditDate: 2018-06-18T19:49:19.950Z
Answers.UserDisplayName: Alex Reynolds
Answers.Body: <p>You can use <code>corr.test()</code> from psych <a href="https://www.rdocumentation.org/packages/psych/versions/1.8.4/topics/corr.test" rel="nofollow noreferrer">package</a>, e.g. </p>
  
  <pre><code>ct &lt;- corr.test(t(A.data), t(B.data), method='spearman', adjust='none')
  </code></pre>
  
  <p>then <code>ct$p</code> and <code>ct$r</code> would give you (uncorrected) p-value and correlation matrices of size <code>795 x 925</code>. Note that <code>t()</code> is required to compare matrices row-wise.</p>
  
  <p>That would be orders of magnitude faster than loop-based approach.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-17T12:35:37.597Z
Answers.LastActivityDate: 2018-06-17T12:35:37.597Z
Answers.UserDisplayName: gkcn
Body: <p>I have two datasets (DataA and DataB) and I want to find the Spearman correlation between genes and also pull out the gene names (stored in first column of dataset) in <code>R</code>. I am using <code>fread</code> from <code>read.table</code> to read the file and <code>cor.test</code> to find <code>Rho</code> and <code>p-value</code>.
  The code works fine for a smaller data set of 10 obs, 10 variables. But, my actual data set is larger (DataA - 795 obj, 542 variables and DataB - 925 obs, 542 variables). 
  When I try to run the code on my actual data, it is consuming a large amount of time (it has been 23 hrs since my code is running). Is there a way to optimize this code? Or is there a bug in the code?</p>
  
  <pre><code>library(data.table)
  A.data &lt;- fread("DataA.csv", header = TRUE, data.table = FALSE, stringsAsFactors = FALSE)
  B.data &lt;- fread("DataB.csv", header = TRUE, data.table = FALSE, stringsAsFactors = FALSE)
  
  correlation.result &lt;- data.frame()
  
  ## Computing Correlation
  
  for (i in 1:nrow(A.data)) {
    for (j in 1:nrow(B.data)) {
         correln &lt;- cor.test(as.numeric(A.data[i,2:ncol(A.data)]), as.numeric(B.data[j,2:ncol(B.data)]), method = "spearman", exact = FALSE)
         rho &lt;- correln$estimate
         p.val &lt;- correln$p.value
         Gene &lt;- paste(A.data[i,1], B.data[j,1], sep="_", collapse=" ") # To pullout gene name pairwise
         comb.data &lt;- data.frame(rho, p.val, Gene)
         correlation.result &lt;- rbind(correlation.result,comb.data)
   } 
  }
  </code></pre>
  
Tags: <r>
Title: Spearman correlation for large dataset
ThreadText: # Spearman correlation for large dataset
  
  (Asked by: username_0 on 2018-06-16)
  
  I have two datasets (DataA and DataB) and I want to find the Spearman correlation between genes and also pull out the gene names (stored in first column of dataset) in `R`. I am using `fread` from `read.table` to read the file and `cor.test` to find `Rho` and `p-value`.
  The code works fine for a smaller data set of 10 obs, 10 variables. But, my actual data set is larger (DataA - 795 obj, 542 variables and DataB - 925 obs, 542 variables).
  When I try to run the code on my actual data, it is consuming a large amount of time (it has been 23 hrs since my code is running). Is there a way to optimize this code? Or is there a bug in the code?
  
  ```
  library(data.table)
  A.data <- fread("DataA.csv", header = TRUE, data.table = FALSE, stringsAsFactors = FALSE)
  B.data <- fread("DataB.csv", header = TRUE, data.table = FALSE, stringsAsFactors = FALSE)
  
  correlation.result <- data.frame()
  
  ## Computing Correlation
  
  for (i in 1:nrow(A.data)) {
    for (j in 1:nrow(B.data)) {
         correln <- cor.test(as.numeric(A.data[i,2:ncol(A.data)]), as.numeric(B.data[j,2:ncol(B.data)]), method = "spearman", exact = FALSE)
         rho <- correln$estimate
         p.val <- correln$p.value
         Gene <- paste(A.data[i,1], B.data[j,1], sep="_", collapse=" ") # To pullout gene name pairwise
         comb.data <- data.frame(rho, p.val, Gene)
         correlation.result <- rbind(correlation.result,comb.data)
   } 
  }
  
  ```
  
  **Tags:** r
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-06-16. Score: 3 | Accepted answer
  
  Your code looks fine, but `rbind` to progressively build and resize a data frame can be gluttonous in terms of memory and time, especially given the A-row x B-row times you are doing this operation, which is fine for very small datasets, but not often realistic for real-world data.
  
  Instead, perhaps `cat` results to a text file, where results are written in tab-delimited columns. You can then read in that text file later on.
  
  In addition to serializing or saving your results to text, which make them easier to work with in non-R contexts, you won't have to deal with R's memory allocation problems when running `rbind` repeatedly. You just write out the answer to a file and read it in later on, in one pass.
  
  Another approach is to preallocate memory to a presized data frame, and then write to that allocated space.
  
  You know the size of your inputs ahead of time, which makes this approach tenable. In this case, it looks like you are storing two floats and a string (`rho`, `p.val`, and `Gene`) and that you are going to store A-row x B-row number of them.
  
  Here's one way you might go about this:
  
  ```
  # create a presized data frame
  Arows <- nrow(A.data)
  Brows <- nrow(B.data)
  correlation.result <- data.frame(t(matrix(c(0.0, 0.0, "foo"), nrow=3, ncol=Arows * Brows)), stringsAsFactors=FALSE)
  colnames(correlation.result) <- c("rho", "p.val", "Gene")
  
  # assign values to that presized data frame 
  Acols <- ncol(A.data)
  Bcols <- ncol(B.data)
  for (i in 1:Arows) {
    # get A data
    Ai <- as.numeric(A.data[i, 2:Acols])
    Aname <- A.data[i, 1]
    for (j in 1:Brows) {
      # calculate row offset in presized data frame
      ij <- (i - 1) * Brows + j
      # get B data
      Bj <- as.numeric(B.data[j, 2:Bcols])
      Bname <- B.data[j, 1]
      # run test
      correln <- cor.test(Ai, Bj, method = "spearman", exact = FALSE)
      rho <- correln$estimate
      p.val <- correln$p.value
      Gene <- paste(Aname, Bname, sep="_", collapse=" ")
      # assign value to row of result data frame
      correlation.result[ij,] <- c(rho, p.val, Gene)
    }
  }
  
  ```
  
  This avoids R needing to resize and recopy the data frame contents as they grow, which I suspect is the major reason it is taking so long to run.
  
  Chapter or "Circle" 2 of Patrick Burns' book *The R Inferno* discusses the problems with growing objects with `rbind` and ways to handle it — you might find this a useful read. This book is a PDF download via your search engine of choice.
  
  > **Comments:**
  > **username_1**: Is there a reason for the downvote(s)?  
  > **username_1**: I think it is bad for a community when downvoting without any explanation. Please explain yourself.  
  
  ---
  
  ### Answer by username_2 on 2018-06-17. Score: 1
  
  You can use `corr.test()` from psych [package](https://www.rdocumentation.org/packages/psych/versions/1.8.4/topics/corr.test), e.g.
  
  ```
  ct <- corr.test(t(A.data), t(B.data), method='spearman', adjust='none')
  
  ```
  
  then `ct$p` and `ct$r` would give you (uncorrected) p-value and correlation matrices of size `795 x 925`. Note that `t()` is required to compare matrices row-wise.
  
  That would be orders of magnitude faster than loop-based approach.
  
  
  ---
  
  