title: tidyr::spread() with plain Scala and Spark (rows to columns)
body: <p>my prototype (written in R with the packages <a href="https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html" rel="nofollow">dplyr</a> and <a href="https://cran.r-project.org/web/packages/tidyr/index.html" rel="nofollow">tidyr</a>) is hitting a wall in terms of computational complexity - even on my powerfull working station. Therefore, I want to port the code to <a href="https://spark.apache.org/" rel="nofollow">Spark</a> using Scala.</p>
  
  <p>I looked up all <a href="http://spark.apache.org/docs/latest/programming-guide.html#actions" rel="nofollow">transformations</a>, <a href="http://spark.apache.org/docs/latest/programming-guide.html#actions" rel="nofollow">actions</a>, <a href="http://spark.apache.org/docs/2.0.0/api/scala/index.html#org.apache.spark.sql.functions$" rel="nofollow">functions</a> (SparkSQL) and <a href="http://spark.apache.org/docs/2.0.0/api/scala/index.html#org.apache.spark.sql.Column" rel="nofollow">column operations</a> (also SparkSQL) and found all function equivalents except the one for the <code>tidyr::spread()</code> function, available in R. </p>
  
  <p><code>df %&gt;% tidyr::spread(key = COL_KEY , value = COL_VAL)</code> basically spreads a key-value pair across multiple columns. E.g. the table</p>
  
  <pre><code>COL_KEY | COL_VAL
  -----------------
  A       | 1
  B       | 1
  A       | 2
  </code></pre>
  
  <p>will be transformed to by </p>
  
  <pre><code>A       | B
  ------------
  1       | 0
  0       | 1
  2       | 1
  </code></pre>
  
  <p>In case there is no "out-of-the-box"-solution available: Could you point me in the right direction? Maybe a user defined function?</p>
  
  <p>I'm free which Spark (and Scala) version to choose (therefore I'd go for the latest, <code>2.0.0</code>).</p>
  
  <p>Thanks!</p>