title: Append a new column to an existing parquet file
body: <p>Is there any way to append a new column to an existing parquet file?</p>
  
  <p>I'm currently working on a kaggle competition, and I've converted all the data to parquet files. </p>
  
  <p>Here was the case, I read the parquet file into pyspark DataFrame, did some feature extraction and appended new columns to DataFrame with </p>
  
  <blockquote>
    <p>pysaprk.DataFrame.withColumn().</p>
  </blockquote>
  
  <p>After that, I want to save the new columns in the source parquet file.</p>
  
  <p>I know Spark SQL come with <a href="https://spark.apache.org/docs/1.4.0/sql-programming-guide.html#schema-merging" rel="noreferrer">Parquet schema evolution</a>, but the example only have shown the case with a key-value.</p>
  
  <p>The parquet "append" mode doesn't do the trick either. It only append new rows to the parquet file. 
  If there's anyway to append a new column to an existing parquet file instead of generate the whole table again? 
  Or I have to generate a separate new parquet file and join them on the runtime. </p>