title: Run 3000+ Random Forest Models By Group Using Spark MLlib Scala API
body: <p>I am trying to build random forest models by group(School_ID, more than 3 thousands) on a large model input csv file using Spark Scala API. Each of the group contains about 3000-4000 records. The resources I have at disposal are 20-30 aws m3.2xlarge instances. </p>
  
  <p>In R, I can construct models by group and save them to a list like this- </p>
  
  <pre><code>library(dplyr);library(randomForest);
      Rf_model &lt;- train %&gt;% group_by(School_ID) %&gt;% 
                  do(school= randomForest(formula=Rf_formula, data=., importance = TRUE))
  </code></pre>
  
  <p>The list can be stored somewhere and I can call them when I need to use them like below - </p>
  
  <pre><code>save(Rf_model.school,file=paste0(Modelpath,"Rf_model.dat"))
  load(file=paste0(Modelpath,"Rf_model.dat"))
  pred &lt;-  predict(Rf_model.school$school[school_index][[1]], newdata=test)
  </code></pre>
  
  <p>I was wondering how to do that in Spark, whether or not I need to split the data by group first and how to do it efficiently if it's necessary.</p>
  
  <p>I was able to split up the file by School_ID based on the below code but it seems it creates one individual job to subset for each iteration and takes a long time to finish the jobs. Is there a way to do it in one pass?</p>
  
  <pre><code>model_input.cache()
  
  val schools = model_input.select("School_ID").distinct.collect.flatMap(_.toSeq)
  val bySchoolArray = schools.map(School_ID =&gt; model_input.where($"School_ID" &lt;=&gt; School_ID))
  
  for( i &lt;- 0 to programs.length - 1 ){
    bySchoolArray(i).
      write.format("com.databricks.spark.csv").
      option("header", "true").
      save("model_input_bySchool/model_input_"+ schools(i))
  }
  </code></pre>
  
  <p>Source:
  <a href="https://stackoverflow.com/questions/31669308/how-can-i-split-a-dataframe-into-dataframes-with-same-column-values-in-scala-and">How can I split a dataframe into dataframes with same column values in SCALA and SPARK</a>  </p>
  
  <p><strong>Edit 8/24/2015</strong> 
  I'm trying to convert my dataframe into a format that is accepted by the random forest model. I'm following the instruction on this thread 
  <a href="https://stackoverflow.com/questions/31028806/how-to-create-correct-data-frame-for-classification-in-spark-ml">How to create correct data frame for classification in Spark ML</a></p>
  
  <p>Basically, I create a new variable "label" and store my class in Double. Then I combine all my features using VectorAssembler function and transform my input data as follows-</p>
  
  <pre><code>val assembler = new VectorAssembler().
    setInputCols(Array("COL1", "COL2", "COL3")).
    setOutputCol("features")
  
  val model_input = assembler.transform(model_input_raw).
    select("SCHOOL_ID", "label", "features")
  </code></pre>
  
  <p>Partial error message(let me know if you need the complete log message) - </p>
  
  <blockquote>
    <p>scala.MatchError: StringType (of class
    org.apache.spark.sql.types.StringType$)
            at org.apache.spark.ml.feature.VectorAssembler$$anonfun$2.apply(VectorAssembler.scala:57)</p>
  </blockquote>
  
  <p>This is resolved after converting all the variables to numeric types.</p>
  
  <p><strong>Edit 8/25/2015</strong>
  The ml model doesn't accept the label I coded manually so I need to use StringIndexer to go around the problem as indicated <a href="https://stackoverflow.com/questions/32108887/how-to-use-randomforest-in-spark-pipeline">here</a>. According to the <a href="https://spark.apache.org/docs/latest/ml-features.html#stringindexer" rel="nofollow noreferrer">official documentation</a>, the most frequent label gets 0. It causes inconsistent labels across School_ID. I was wondering if there's a way to create the labels without resetting the order of the values.</p>
  
  <pre><code>val indexer = new StringIndexer().
    setInputCol("label_orig").
    setOutputCol("label")
  </code></pre>
  
  <p>Any suggestions or directions would be helpful and feel free to raise any questions. Thanks!</p>