title: Apply a function to a single column of a csv in Spark
body: <p>Using Spark I'm reading a csv and want to apply a function to a column on the csv. I have some code that works but it's very hacky. What is the proper way to do this?</p>
  
  <p>My code</p>
  
  <pre><code>SparkContext().addPyFile("myfile.py")
  spark = SparkSession\
      .builder\
      .appName("myApp")\
      .getOrCreate()
  from myfile import myFunction
  
  df = spark.read.csv(sys.argv[1], header=True,
      mode="DROPMALFORMED",)
  a = df.rdd.map(lambda line: Row(id=line[0], user_id=line[1], message_id=line[2], message=myFunction(line[3]))).toDF()
  </code></pre>
  
  <p>I would like to be able to just call the function on the column name instead of mapping each row to <code>line</code> and then calling the function on <code>line[index]</code>.</p>
  
  <p>I'm using Spark version 2.0.1</p>