Body: <p>I have a vector of 200 million numbers. It has been running for over 24 hours and still not done. </p>
  
  <pre><code>x &lt;- data   # the target sequence
  k &lt;- exp(-(-win:win)^2/(2*sig^2))/(sig*sqrt(2*pi))  # gaussian kernel
  ptm &lt;- proc.time()
  y &lt;- convolve(x,k,type="open")
  proc.time()-ptm
  </code></pre>
  
  <p>How to parallelize the <a href="https://www.rdocumentation.org/packages/stats/versions/3.6.1/topics/convolve" rel="nofollow noreferrer">convolve</a> function?
  To parallelize, I tried splitting up the vector into pieces:</p>
  
  <pre><code>&gt; x &lt;- data[1:1e7]
  &gt; ptm &lt;- proc.time()
  &gt; y &lt;- convolve(x,k,type="open") # smoothed sequence
  &gt; proc.time()-ptm
     user  system elapsed 
   40.743   0.206  40.940 
  &gt; x &lt;- data[1e7+1:2e7]
  &gt; ptm &lt;- proc.time()
  &gt; y &lt;- convolve(x,k,type="open") # smoothed sequence
  &gt; proc.time()-ptm
     user  system elapsed 
  320.193   0.708 320.871 
  </code></pre>
  
  <p>However, splitting it up and then adding back the convolutions will be tricky. 
  Does anyone have other ideas?</p>
  
  <p>This data is based on biological data. We want to cluster values in the vector based on nearby vector indices.</p>
  
  <p>Reproducible example with 20,000,000 random numbers should finish in about 5.5-6 minutes. But if you try running 200,000,000 locally or on HPC, it will take a long time (my HPC has limit 99 hours and it timed out at 99 hours unfinished).</p>
  
  <pre><code>win &lt;- 100        # window length of gaussian kernel (units acc to orig sequence)
  sig &lt;- 5        # std deviation of gaussian
  k &lt;- exp(-(-win:win)^2/(2*sig^2))/(sig*sqrt(2*pi))  # gaussian kernel
  l &lt;- 20000000
  x &lt;- round(runif(l, 0.0, 1.0),1)   # the target sequence with peaks
  t &lt;- seq(0,1-1/length(x),1/length(x))
  ptm &lt;- proc.time()
  y &lt;- convolve(x,k,type="open")[(win+1):(win+length(x))] # smoothed sequence
  proc.time()-ptm
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-31T15:38:53.633Z
Comments.Text: Is the function vectorized? Did you try the same function with less numbers? If you need to make a cluster, why do you need to make the convolution? Perhaps you will get the attention of those that could help you better on stackoverflow.com.
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-03T13:14:45.453Z
Comments.Text: Hi IIrs: The convolve function is a function in R. The input data (x <- data) is a vector. We need to group based on nearby indices in the vector, so we cannot use other clustering methods that were built, for example, clustering gene expression. I can re-post in stackoverflow, but I have heard re-posting the same question is not good (and I am new to the stackoverflow community). I can try re-posting though. Thank you.
Comments.UserDisplayName: user11885521
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-03T15:07:44.577Z
Comments.Text: What kind of vector is it? A list or a numeric one? Could you please post a minimal reproducible example (with faked data if needed)?
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-03T15:59:23.163Z
Comments.Text: Hi IIrs, It is a numeric vector. I've troubleshooted and number of decimal places does not affect time to finish. I have edited the post above with a reproducible example with 20,000,000 random numbers. The time to finish with random numbers matches the time to finish with subsetted real data.
Comments.UserDisplayName: user11885521
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-03T18:56:12.743Z
Comments.Text: I don't know what is supposed to do but maybe the problem is that there isn't any continuous periodicity to use fft on it??
Comments.UserDisplayName: llrs
Tags: <r>
Title: How to run function "convolve" in parallel in R?
ThreadText: # How to run function "convolve" in parallel in R?
  
  (Asked by: username_0 on 2019-08-30)
  
  I have a vector of 200 million numbers. It has been running for over 24 hours and still not done.
  
  ```
  x <- data   # the target sequence
  k <- exp(-(-win:win)^2/(2*sig^2))/(sig*sqrt(2*pi))  # gaussian kernel
  ptm <- proc.time()
  y <- convolve(x,k,type="open")
  proc.time()-ptm
  
  ```
  
  How to parallelize the [convolve](https://www.rdocumentation.org/packages/stats/versions/3.6.1/topics/convolve) function?
  To parallelize, I tried splitting up the vector into pieces:
  
  ```
  > x <- data[1:1e7]
  > ptm <- proc.time()
  > y <- convolve(x,k,type="open") # smoothed sequence
  > proc.time()-ptm
     user  system elapsed 
   40.743   0.206  40.940 
  > x <- data[1e7+1:2e7]
  > ptm <- proc.time()
  > y <- convolve(x,k,type="open") # smoothed sequence
  > proc.time()-ptm
     user  system elapsed 
  320.193   0.708 320.871 
  
  ```
  
  However, splitting it up and then adding back the convolutions will be tricky.
  Does anyone have other ideas?
  
  This data is based on biological data. We want to cluster values in the vector based on nearby vector indices.
  
  Reproducible example with 20,000,000 random numbers should finish in about 5.5-6 minutes. But if you try running 200,000,000 locally or on HPC, it will take a long time (my HPC has limit 99 hours and it timed out at 99 hours unfinished).
  
  ```
  win <- 100        # window length of gaussian kernel (units acc to orig sequence)
  sig <- 5        # std deviation of gaussian
  k <- exp(-(-win:win)^2/(2*sig^2))/(sig*sqrt(2*pi))  # gaussian kernel
  l <- 20000000
  x <- round(runif(l, 0.0, 1.0),1)   # the target sequence with peaks
  t <- seq(0,1-1/length(x),1/length(x))
  ptm <- proc.time()
  y <- convolve(x,k,type="open")[(win+1):(win+length(x))] # smoothed sequence
  proc.time()-ptm
  
  ```
  
  **Tags:** r
  
  > **Comments:**
  > **username_1**: Is the function vectorized? Did you try the same function with less numbers? If you need to make a cluster, why do you need to make the convolution? Perhaps you will get the attention of those that could help you better on stackoverflow.com.  
  > **username_0**: Hi IIrs: The convolve function is a function in R. The input data (x <- data) is a vector. We need to group based on nearby indices in the vector, so we cannot use other clustering methods that were built, for example, clustering gene expression. I can re-post in stackoverflow, but I have heard re-posting the same question is not good (and I am new to the stackoverflow community). I can try re-posting though. Thank you.  
  > **username_1**: What kind of vector is it? A list or a numeric one? Could you please post a minimal reproducible example (with faked data if needed)?  
  > **username_0**: Hi IIrs, It is a numeric vector. I've troubleshooted and number of decimal places does not affect time to finish. I have edited the post above with a reproducible example with 20,000,000 random numbers. The time to finish with random numbers matches the time to finish with subsetted real data.  
  > **username_1**: I don't know what is supposed to do but maybe the problem is that there isn't any continuous periodicity to use fft on it??  
  
  ---
  
  ## 0 answers
  
  