title: Apply Model Scores to Spark DataFrame - Python
body: <p>I'm trying to apply a score to a Spark DataFrame using PySpark.  Let's assuming that I built a simple regression model outside of Spark and want to map the coefficient values created in the model to the individual columns in the DataFrame to create a new column that is the sum of each of the different source columns multiplied by the individual coefficients.  I understand that there are many utilities in Spark mllib for modeling, but I want to understand how this 'brute force' method could be accomplished.  I also know that DataFrames/RDDs are immutable, so a new DataFrame would have to be created. </p>
  
  <p>Here's some pseudo-code for reference:</p>
  
  <pre><code>#load example data
  df = sqlContext.createDataFrame(data)
  df.show(5)
  dfmappd.select("age", "parch", "pclass").show(5)
  +----+-----+------+
  | age|parch|pclass|
  +----+-----+------+
  |22.0|    0|     3|
  |38.0|    0|     1|
  |26.0|    0|     3|
  |35.0|    0|     1|
  |35.0|    0|     3|
  +----+-----+------+
  only showing top 5 rows
  </code></pre>
  
  <p>The model created outside of Spark is a logistic regression model based on a binary response.  So essentially I want to map the logit function to these three columns to produce a fourth scored column.  Here are the coefficients from the model:</p>
  
  <p>intercept: 3.435222</p>
  
  <p>age: -0.039841</p>
  
  <p>parch: 0.176439</p>
  
  <p>pclass: -1.239452</p>
  
  <p>Here is a description of the logit function for reference:
  <a href="https://en.wikipedia.org/wiki/Logistic_regression" rel="nofollow">https://en.wikipedia.org/wiki/Logistic_regression</a></p>
  
  <p>For comparison, here is how I would do the same thing in R using tidyr and dplyr</p>
  
  <pre><code>library(dplyr)
  library(tidyr)
  
  #Example data
  Age &lt;- c(22, 38, 26, 35, 35)
  Parch &lt;- c(0,0,0,0,0)
  Pclass &lt;- c(3, 1, 3, 1, 3)
  
  #Wrapped in a dataframe
  mydf &lt;- data.frame(Age, Parch, Pclass)
  
  #Using dplyr to create a new dataframe with mutated column
  scoredf = mydf %&gt;% 
    mutate(score = round(1/(1 + exp(-(3.435 + -0.040 * Age + 0.176 * Parch + -1.239 * Pclass))),2))
  scoredf
  </code></pre>