title: Memory issue with foreach loop in R on Windows 8 (64-bit) (doParallel package)
body: <p>I'm trying to move from a serial to parallel approach to accomplish some multivariate time series analysis tasks on a large <code>data.table</code>.  The table contains data for many different groups and I'm trying to move from a <code>for</code> loop to a <code>foreach</code> loop using the <code>doParallel</code> package to take advantage of the multicore processor installed.</p>
  
  <p>The problem I am experiencing relates to memory and how the new R processes seem to consume large quantities of it.  I think that what is happening is that the large <code>data.table</code> containing ALL data is copied into each new process, hence I run out of RAM and Windows starts swapping to disk.</p>
  
  <p>I've created a simplified reproducible example which replicates my problem, but with less data and less analysis inside the loop.  It would be ideal if a solution existed which could only farm out the data to the worker processes on demand, or sharing the memory already used between cores.  Alternatively some kind of solution may already exist to split the big data into 4 chunks and pass these to the cores so they have a subset to work with.</p>
  
  <p>A similar question has previously been posted <a href="https://stackoverflow.com/questions/18028452/reading-global-variables-using-foreach-in-r?lq=1">here on Stackoverflow</a> however I cannot make use of the <code>bigmemory</code> solution offered as my data contains a character field.  I will look further into the <code>iterators</code> package, however I'd appreciate any suggestions from members with experience of this problem in practice.</p>
  
  <pre><code>rm(list=ls())
  library(data.table)
  num.series = 40    # can customise the size of the problem (x10 eats my RAM)
  num.periods = 200  # can customise the size of the problem (x10 eats my RAM)
  dt.all = data.table(
      grp     = rep(1:num.series,each=num.periods), 
      pd      = rep(1:num.periods, num.series), 
      y       = rnorm(num.series * num.periods),
      x1      = rnorm(num.series * num.periods),
      x2      = rnorm(num.series * num.periods)
  ) 
  dt.all[,y_lag := c(NA, head(y, -1)), by = c("grp")]
  
  f_lm = function(dt.sub, grp) {
      my.model = lm("y ~ y_lag + x1 + x2 ", data = dt.sub)
      coef = summary(my.model)$coefficients
      data.table(grp, variable = rownames(coef), coef)
  }
  
  library(doParallel)  
  registerDoParallel(4) 
  foreach(grp=unique(dt.all$grp), .packages="data.table", .combine="rbind")  %dopar%
  {
      dt.sub = dt.all[grp == grp]
      f_lm(dt.sub, grp)
  }
  detach(package:doParallel)
  </code></pre>