title: Spark UDF not working: how to specify the column on which to apply it?
body: <p>Let's say I have my DataFrame, with a given column named "X". I want to understand why the first code doesn't work whereas the second one does. For me, it doesn't change anything.</p>
  
  <p>On the one hand, this doesn't work:</p>
  
  <pre><code>val dataDF = sqlContext
        .read
        .parquet(input_data)
        .select(
            "XXX", "YYY", "III"
        )
        .toDF(
            "X", "Y", "I"
        )
        .groupBy(
            "X", "Y"
        )
        .agg(
            sum("I").as("sum_I")
        )
        .orderBy(desc("sum_I"))
        .withColumn("f_sum_I", udf((x: Long) =&gt; f(x)).apply(dataDF("sum_I")))
        .drop("sum_I")
  
  dataDF.show(50, false)
  </code></pre>
  
  <p>IntelliJ doesn't compile my code and I have the following error: </p>
  
  <pre><code>Error:(88, 67) recursive value dataDF needs type
        .withColumn("f_sum_I", udf((x: Long) =&gt; f(x)).apply(dataDF("sum_I")))
  </code></pre>
  
  <p>On the other hand, this work if I change the given line with this:</p>
  
  <pre><code>.withColumn("f_sum_I", udf((x: Long) =&gt; f(x)).apply(col("sum_I")))
  </code></pre>
  
  <p>All I did was replacing the call to my DataFrame column to use a more generic function "col". I don't understand the difference, and more especially why does it not prefer the first method (with the name of the DataFrame).</p>