title: How to setup workers for parallel processing in R using snowfall and multiple Windows nodes?
body: <p>I’ve successfully used snowfall to setup a cluster on a single server with 16 processors.</p>
  
  <pre><code>require(snowfall)
  if (sfIsRunning() == TRUE) sfStop()
  
  number.of.cpus &lt;- 15
  sfInit(parallel = TRUE, cpus = number.of.cpus)
  stopifnot( sfCpus() == number.of.cpus )
  stopifnot( sfParallel() == TRUE )
  
  # Print the hostname for each cluster member
  sayhello &lt;- function()
  {
      info &lt;- Sys.info()[c("nodename", "machine")]
      paste("Hello from", info[1], "with CPU type", info[2])
  }
  names &lt;- sfClusterCall(sayhello)
  print(unlist(names))
  </code></pre>
  
  <p>Now, I am looking for complete instructions on <strong>how to move to a distributed model</strong>. I have 4 different Windows machines with a total of 16 cores that I would like to use for a 16 node cluster. So far, I understand that I could manually setup a SOCK connection or leverage MPI. While it appears possible, I haven’t found clear and complete directions as to how.</p>
  
  <p><strong>The SOCK route</strong> appears to depend on code in a snowlib script. I can generate a stub from the master side with the following code:</p>
  
  <pre><code>winOptions &lt;-
      list(host="172.01.01.03",
           rscript="C:/Program Files/R/R-2.7.1/bin/Rscript.exe",
           snowlib="C:/Rlibs")
  
  cl &lt;- makeCluster(c(rep(list(winOptions), 2)), type = "SOCK", manual = T)
  </code></pre>
  
  <p>It yields the following:</p>
  
  <pre><code>Manually start worker on 172.01.01.03 with
       "C:/Program Files/R/R-2.7.1/bin/Rscript.exe"
        C:/Rlibs/snow/RSOCKnode.R
        MASTER=Worker02 PORT=11204 OUT=/dev/null SNOWLIB=C:/Rlibs
  </code></pre>
  
  <p>It feels like a reasonable start. I found code for <code>RSOCKnode.R</code> on GitHub under the snow package:</p>
  
  <pre><code>local({
      master &lt;- "localhost"
      port &lt;- ""
      snowlib &lt;- Sys.getenv("R_SNOW_LIB")
      outfile &lt;- Sys.getenv("R_SNOW_OUTFILE") ##**** defaults to ""; document
  
      args &lt;- commandArgs()
      pos &lt;- match("--args", args)
      args &lt;- args[-(1 : pos)]
      for (a in args) {
          pos &lt;- regexpr("=", a)
          name &lt;- substr(a, 1, pos - 1)
          value &lt;- substr(a,pos + 1, nchar(a))
          switch(name,
                 MASTER = master &lt;- value,
                 PORT = port &lt;- value,
                 SNOWLIB = snowlib &lt;- value,
                 OUT = outfile &lt;- value)
      }
  
      if (! (snowlib %in% .libPaths()))
          .libPaths(c(snowlib, .libPaths()))
      library(methods) ## because Rscript as of R 2.7.0 doesn't load methods
      library(snow)
  
      if (port == "") port &lt;- getClusterOption("port")
  
      sinkWorkerOutput(outfile)
      cat("starting worker for", paste(master, port, sep = ":"), "\n")
      slaveLoop(makeSOCKmaster(master, port))
  })
  </code></pre>
  
  <p>It’s not clear how to actually start a SOCK listener on the workers, unless it is buried in <code>snow::recvData</code>.</p>
  
  <p>Looking into <strong>the MPI route</strong>, as far as I can tell, Microsoft MPI version 7 is a starting point. However, I could not find a Windows alternative for sfCluster. I was able to start the MPI service, but it does not appear to listen on port 22 and no amount of bashing against it with <code>snowfall::makeCluster</code> has yielded a result. I’ve disabled the firewall and tried testing with makeCluster and directly connecting to the worker from the master with PuTTY.</p>
  
  <hr>
  
  <p>Is there a comprehensive, step-by-step guide to setting up a snowfall cluster on Windows workers that I’ve missed? I am fond of <code>snowfall::sfClusterApplyLB</code> and would like to continue using that, but if there is an easier solution, I’d be willing to change course. Looking into Rmpi and parallel, I found alternative solutions for the master side of the work, but still little to no specific detail on how to setup workers running Windows.</p>
  
  <p>Due to the nature of the work environment, neither moving to AWS, nor Linux is an option.</p>
  
  <p>Related questions without definitive answers for Windows worker nodes:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/q/22605238">How to set up cluster slave nodes (on Windows)</a></li>
  <li><a href="https://stackoverflow.com/q/17547994">Parallel R on a Windows cluster</a></li>
  <li><a href="https://stackoverflow.com/q/15039039">Create a cluster of co-workers' Windows 7 PCs for parallel processing in R?</a></li>
  </ul>