title: Parallel computing with foreach - Saving loop outputs to global list
body: <p>I would like to run a large loop with the <code>foreach</code> function.This means using the <code>%dopar%</code> operator.</p>
  
  <p>I can't find any questions already answered to this problem exactly. If this is a duplicate though, please point me in the right direction and I'll close this question.</p>
  
  <p>I have been having mixed success. It works for simple examples on my machine, as per the <a href="https://cran.r-project.org/web/packages/foreach/index.html" rel="nofollow">help documentation</a>, however I cannot seem to get good results for my own work.
  My example is slightly more complicated, so the devil seems to be in the detail, as always! I have also read the 'white paper' provided by the package creators Revolution Analytics (<a href="http://www.revolutionanalytics.com/whitepaper/using-foreach-package-r-combine-iterators-and-other-functions" rel="nofollow">you can get it here</a>).
  I don't see how best to maybe use the <code>.combine</code> argument to apply results to my global output list.
  I would like to assign claculated value to one big list as opposed to using <code>cbind</code> or <code>c</code></p>
  
  <p>My example is pretty convoluted, but if I simplify it any further then any answers might not address my issue.</p>
  
  <p>I will perform a kind of moving-linear model. So fit a model using lm() over 50 obersvations [1:50], predict the 51st observation [51], saving the results to a list.
  Then I will shift it all one observation further. So a <code>lm</code> over [2:51] and predict the 52nd observation [52].
  I will use a total of 100 observations, so I can make a maximum of 50 predictions.</p>
  
  <pre><code>## ============================================ ##
  ##  Setup the backend for the foreach function  ##
  ## ============================================ ##
  
  ## doMC calls upon cores on demand, uses them and closes them
  library(doMC)
  registerDoMC(detectCores())     #detectCores() uses all cores
  
  ## for Windows users
  #library(doParallel) --&gt; for Windows users
  #registerDoParallel(detectCores())
  
  ## ======================== ##
  ##  Create some dummy data  ##
  ## ======================== ##
  
  ## three columns, one hundred observations
  my_data &lt;- data.table(outcome = runif(100), V1 = 3*runif(100), V2 = sqrt(runif(100)))
  
  ## Have a look at the data if you like - using the DT package
  library(DT)
  datatable(my_data, options = list(pageLength = nrow(my_data)))
  
  ## ================================= ##
  ##  Perform the loop the normal way  ##
  ## ================================= ##
  
  ## Create container (a list of lists) for results
  my_results &lt;- sapply(c(paste0("step_", seq(1:50))), function(x) NULL)
  step_results &lt;- sapply(c("coefs", "rank", "error"), function(x) NULL)
  for(i in 1:length(my_results)){my_results[[i]] &lt;- step_results}
  
  ## Use a for loop to stpe through all the 50 'slices'
  for(i in 1:50) {        #max. 50 predictions possible
  
      ## Fit a linear model
      my_fit &lt;- lm("outcome ~ V1 + V2", data = my_data[i:(i+49)])
  
      ## Predict the next step
      my_pred &lt;- predict(my_fit, newdata = my_data[i+50, .(V1, V2)]) 
  
      error &lt;- my_data$outcome[i+50] - my_pred    #simply measure the delta to the actual value
  
      ## Assign some results to the container created earlier
      my_results[[i]][[1]] &lt;- my_fit$coefficients
      my_results[[i]][[2]] &lt;- my_fit$rank
      my_results[[i]][[3]] &lt;- error
  
  }
  str(my_results)    ## Keep this container to compare to our next one
  
  ## ============================================ ##
  ##  Perform the loop using foreach and %dopar%  ##
  ## ============================================ ##
  
  ## Create same results object for results as previously for parallel results
  par_results &lt;- sapply(c(paste0("step_", seq(1:50))), function(x) NULL)
  step_results &lt;- sapply(c("coefs", "rank", "error"), function(x) NULL)
  for(i in 1:length(par_results)){par_results[[i]] &lt;- step_results}
  
  my_results_par &lt;- foreach(i = 1:50) %dopar%
      {        #max. 50 predictions possible
  
          my_fit &lt;- lm("outcome ~ V1 + V2", data = my_data[i:(i+49)])     
          my_pred &lt;- predict(my_fit, newdata = my_data[i+50, .(V1, V2)]) 
          error &lt;- my_data$outcome[i+50] - my_pred 
  
          ## Assign some results to the container created earlier
          par_results[[i]][[1]] &lt;- my_fit$coefficients
          par_results[[i]][[2]] &lt;- my_fit$rank
          par_results[[i]][[3]] &lt;- error
  
          Sys.sleep(i/20)    #Allows time to see R processes spawn on your system
          return(par_results)
      }
  
  ## We can see straight away that this didn't work as I would like it to
  identical(my_results, my_results_par)   #FALSE
  
  ## This shows that the output seems good on the surface
  class(my_results_par)
  length(my_results_par)
  ## This shows that it doesn't (WARNING: very long)
  str(my_results_par)
  </code></pre>
  
  <p>You can try out the various <code>.combine</code> arguments in the <code>foreach</code> function, for example:</p>
  
  <pre><code>foreach(i = 1:50, .combine = "c") {computation}
  </code></pre>
  
  <p>or</p>
  
  <pre><code>foreach(i = 1:50, .combine = "cbind") {computation}
  </code></pre>
  
  <p>these prodice a vector and a matrix respectively, but do not contain all the results that I was trying to save in each loop.</p>
  
  <p><strong>Questions</strong></p>
  
  <ol>
  <li>Does that structure give you a clue as to what is going on?</li>
  <li>How might I use <code>.combine</code> argument to create my desired output?</li>
  <li>Is what I am trying to do even possible??</li>
  <li>Do I need to put the loop with foreach at a different point in the algorithm?</li>
  </ol>
  
  <p>I have read that you can supply a custom function to <code>foreach</code>... might this be the way to do it? I still don't see how I would combine the results.</p>