Answers.Body: <p>The bad news is that, indeed, projectiveKMeans is not parallelized and I am not sure how much of it is (easily) parallelizable. The good news is that with 15k features (genes) and 96GB RAM you don't need to run preclustering at all. Just analyze the whole data set in one block.</p>
  
  <p>If you for some reason insist on splitting the data into blocks, make sure the BLAS R uses on the cluster is a fast BLAS, ideally one that can parallelize matrix operations. A lot of time in merging of small clusters is spent calculating singular value decompositions, which becomes much faster using a good BLAS. HPC clusters will often have Intel KML available, or the admins could install OpenBLAS.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-24T13:06:41.343Z
Answers.Text: Thanks for each suggestion. Both helpful!  I definitely don't need to run preclustering, makes sense.  Will also make sure we are running Intel KML or OpenBLAS, I suspect this will already have been considered and optimized.
Answers.UserDisplayName: zdebruine
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-24T13:43:20.350Z
Answers.Text: Will WGCNA k-means run using GPU? I have a hunch it might do.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-24T13:49:02.263Z
Answers.Text: Okay I checked and the answer is no. You're stuck, my bad. Reference https://bmcsystbiol.biomedcentral.com/articles/10.1186/s12918-017-0420-6
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-24T06:17:43.733Z
Answers.LastActivityDate: 2019-12-24T06:17:43.733Z
Answers.UserDisplayName: Peter Langfelder
Body: <p>WGCNA carefully takes advantage of server parallelization in several functions, and I can switch this on with:</p>
  
  <pre><code>enableWGCNAThreads(nThreads=28)
  </code></pre>
  
  <p>That speeds up some functions significantly, but the two bottleneck functions in the pipeline (in my experience) are part of the "Projective k-means" operation, primarily <code>"..k-means clustering.."</code> and <code>"..merging smaller clusters.."</code>.</p>
  
  <p>Unfortunately, parallelization does not work for those two functions.</p>
  
  <p>Am I missing something?  How can I parallelize these functions?  Otherwise I'm wasting HPC resources and cluster time.</p>
  
  <p>For reference, I'm working on a 28-processor HPC node with 96GB RAM, and I'm trying to run analysis on a 15,000-square correlation matrix.</p>
  
Tags: <r><wgcna>
Title: Parallelizing WGCNA k-means clustering and merging smaller clusters
ThreadText: # Parallelizing WGCNA k-means clustering and merging smaller clusters
  
  (Asked by: username_0 on 2019-12-24)
  
  WGCNA carefully takes advantage of server parallelization in several functions, and I can switch this on with:
  
  ```
  enableWGCNAThreads(nThreads=28)
  
  ```
  
  That speeds up some functions significantly, but the two bottleneck functions in the pipeline (in my experience) are part of the "Projective k-means" operation, primarily `"..k-means clustering.."` and `"..merging smaller clusters.."`.
  
  Unfortunately, parallelization does not work for those two functions.
  
  Am I missing something? How can I parallelize these functions? Otherwise I'm wasting HPC resources and cluster time.
  
  For reference, I'm working on a 28-processor HPC node with 96GB RAM, and I'm trying to run analysis on a 15,000-square correlation matrix.
  
  **Tags:** r, wgcna
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-24. Score: 2 | Accepted answer
  
  The bad news is that, indeed, projectiveKMeans is not parallelized and I am not sure how much of it is (easily) parallelizable. The good news is that with 15k features (genes) and 96GB RAM you don't need to run preclustering at all. Just analyze the whole data set in one block.
  
  If you for some reason insist on splitting the data into blocks, make sure the BLAS R uses on the cluster is a fast BLAS, ideally one that can parallelize matrix operations. A lot of time in merging of small clusters is spent calculating singular value decompositions, which becomes much faster using a good BLAS. HPC clusters will often have Intel KML available, or the admins could install OpenBLAS.
  
  > **Comments:**
  > **username_0**: Thanks for each suggestion. Both helpful!  I definitely don't need to run preclustering, makes sense.  Will also make sure we are running Intel KML or OpenBLAS, I suspect this will already have been considered and optimized.  
  > **username_2**: Will WGCNA k-means run using GPU? I have a hunch it might do.  
  > **username_2**: Okay I checked and the answer is no. You're stuck, my bad. Reference https://bmcsystbiol.biomedcentral.com/articles/10.1186/s12918-017-0420-6  
  
  ---
  
  