title: PySpark - to_date format from column
body: <p>I am currently trying to figure out, how to pass the String - format argument to the to_date pyspark function via a column parameter.</p>
  
  <p>Specifically, I have the following setup:</p>
  
  <pre class="lang-python prettyprint-override"><code>sc = SparkContext.getOrCreate()
  df = sc.parallelize([('a','2018-01-01','yyyy-MM-dd'),
                        ('b','2018-02-02','yyyy-MM-dd'),
                        ('c','02-02-2018','dd-MM-yyyy')]).toDF(
                      ["col_name","value","format"])
  </code></pre>
  
  <p>I am currently trying to add a new column, where each of the dates from the column F.col("value"), which is a string value, is parsed to a date.</p>
  
  <p>Separately for each format, this can be done with</p>
  
  <pre class="lang-python prettyprint-override"><code>df = df.withColumn("test1",F.to_date(F.col("value"),"yyyy-MM-dd")).\
          withColumn("test2",F.to_date(F.col("value"),"dd-MM-yyyy"))
  </code></pre>
  
  <p>This however gives me 2 new columns - but I want to have 1 column containing both results - but calling the column does not seem to be possible with the to_date function:</p>
  
  <pre class="lang-python prettyprint-override"><code>df = df.withColumn("test3",F.to_date(F.col("value"),F.col("format")))
  </code></pre>
  
  <p>Here an error "Column object not callable" is being thrown.</p>
  
  <p>Is is possible to have a generic approach for all possible formats (so that I do not have to manually add new columns for each format)?</p>