title: Do User Defined Functions (UDF) in Spark Run on Parallel on Cluster Worker Nodes?
body: <p>Let assume I have created a function in python so raised a number to power 2:</p>
  
  <pre><code>def squared(s):
    return s * s
  </code></pre>
  
  <p>And then I registered the function in <code>Spark session</code> as below:</p>
  
  <pre><code>spark.udf.register("squaredWithPython", squared)
  </code></pre>
  
  <p>then when I call the UDF in Spark SQL as in: </p>
  
  <pre><code>spark.range(1, 20).registerTempTable("test")
  %sql select id, squaredWithPython(id) as id_squared from test
  </code></pre>
  
  <p>Then is the function <code>squaredWithPython</code> going to run on the worker nodes of the cluster, if the data is distributed on the workers memory? If yes, then what vectorized <code>UDF</code>s used for? And what is the difference between <code>UDF</code> and <code>vectorized UDF</code>?</p>
  
  <p>Likewise, for the use of <code>UDF</code> with DataFrames.</p>
  
  <p>Please note that the code is retrieved from: <a href="https://docs.databricks.com/spark/latest/spark-sql/udf-python.html" rel="nofollow noreferrer">https://docs.databricks.com/spark/latest/spark-sql/udf-python.html</a> </p>
  
  <p>Any help is much appreciated!!</p>