title: Appending data to a automatic partitioned dataframe stored externally as parquet files
body: <p>Trying to write auto partitioned Dataframe data on an attribute to external store in append mode overwrites the parquet files.</p>
  
  <p>I have a huge amount of data that I cannot load in one go. So I am reading data a folder at a time in a loop. In every iteration, I partition the data based on a certain attribute and use saveAsTable to write the parquet files on Amazon S3. I am finding that my s3 folder is getting wiped out at every iteration. I want to add on the data from every iteration to my hive store in partitioned folders, so I can categorize the data and can read only the category I want to work on.</p>
  
  <p>This is the command I am using to save the dataframe.</p>
  
  <pre><code>DF.write.partitionBy('Type').format('parquet').mode("append").saveAsTable(('AllComponents', path='s3a://xxx/&lt;Path&gt;')
  
  for Pos1 in HexKey: 
      folderKey = "{}".format(Pos1)
  
      spark = SparkSession.builder \
          .getOrCreate()
  
      if DataSetSchema is None:
         log.warn("Reviewing schema")
         AllComponentsDF = spark.read \
           .format('com.databricks.spark.xml') \
           .load('s3a://location' + folderKey + '0/00/*')
         DataSetSchema = AllComponentsDF.schema
      else:
          log.warn("Reading folder {}".format(Pos1))  
          AllComponentsDF = spark.read \
              .format('com.databricks.spark.xml') \
              .load('s3a://location/' + folderKey + '0/00/*', schema=DataSetSchema)
          AllComponentsDF.write.partitionBy('Type').format('parquet').mode("append").saveAsTable(('AllComponents', path='s3a://spark-cluster-boomi/AllComponents')
  </code></pre>