title: Performance difference map() vs withColumn()
body: <p>I have a table with over 100 columns. I need to remove double quotes from certain columns. I found 2 ways to do it, using withColumn() and map()</p>
  <p>Using withColumn()</p>
  <pre><code>cols_to_fix = [&quot;col1&quot;, ..., &quot;col20&quot;]
  for col in cols_to_fix:
      df = df.withColumn(col, regexp_replace(df[col], &quot;\&quot;&quot;, &quot;&quot;))
  </code></pre>
  <p>Using map()</p>
  <pre><code>def remove_quotes(row: Row) -&gt; Row:
      row_as_dict = row.asDict()
      cols_to_fix = [&quot;col1&quot;, ..., &quot;col20&quot;]
      for column in cols_to_fix:
          if row_as_dict[column]:
              row_as_dict[column] = re.sub(&quot;\&quot;&quot;, &quot;&quot;, str(row_as_dict[column]))
      return Row(**row_as_dict)
   
  df = df.rdd.map(remove_quotes).toDF(df.schema)
  </code></pre>
  <p>Here is my question. I found using map() takes about 4 times longer than withColumn() on a table that has ~25M records. I will really appreciate if any fellow stack overflow user can explain the reason for the performance difference, so that I can avoid similar pitfall in future.</p>