title: Pyspark : how do I apply the function to get slope in a dataframe?
body: <p>I have a spark dataframe like this (x and y columns, each with 6 datapoints). I want to be able to extract the slope by fitting a simple regression line (basically to see the trend of y, increasing or decreasing).</p>
  
  <pre><code>
  +------------------+----------------------------------------------------------------------------------------------------------------------------------------+
  |x                 |y                                                                                                                                       |
  +------------------+----------------------------------------------------------------------------------------------------------------------------------------+
  |[1, 2, 3, 4, 5, 6]|[3.42102562348192E-6, 4.2159323917750995E-6, 3.924587540944015E-6, 4.167182871752131E-6, 4.109192066532302E-6, 4.297804458327455E-6]    |
  |[1, 2, 3, 4, 5, 6]|[1.384402399630826E-5, 9.913141993957704E-6, 1.1145077060247102E-5, 1.1005472165326649E-5, 1.1004462921073546E-5, 1.1004462921073546E-5]|
  +------------------+----------------------------------------------------------------------------------------------------------------------------------------+
  </code></pre>
  
  <p>I want to use this function but I am getting errors when applying to dataframe. what is the correct way to apply this function which takes in 2 arrays?</p>
  
  <pre><code>import numpy as np
  def trendline(x,y, order=1):
      coeffs = np.polyfit(x, y, order)
      slope = coeffs[-2]
      return float(slope)
  
  #example to run
  x_example=[1,2,3,4,5,6]
  y_example=[3.42102562348192,4.2159323917750995,3.924587540944015,4.167182871752131,4.109192066532302,4.297804458327455 ]
  slope=trendline(x_example,y_example)
  print(slope)  
  </code></pre>
  
  <p>this functions works itself like the example above but i have trouble applying to the dataframe , i want to create a new column in the dataframe with returned slople.</p>
  
  <p>thank you!!</p>
  
  <p>I tried this and it didn't work</p>
  
  <pre><code>def get_slope_func(x,y, order=1):
      coeffs = np.polyfit(x, y, order)
      slope = coeffs[-2]
      return float(slope)
  get_slope = pandas_udf(get_slope_func, returnType=LongType())
  
  df.select(get_slope(col("x"), col("y"))).show()
  
  </code></pre>