title: How to avoid empty files while writing parquet files?
body: <p>I am reading from Kafka queue using Spark Structured Streaming. After reading from Kafka I am applying <code>filter</code> on the <code>dataframe</code>. I am saving this filtered dataframe into a parquet file. This is generating many empty parquet files. Is there any way I can stop writing an empty file? </p>
  
  <pre class="lang-py prettyprint-override"><code>df = spark \
      .readStream \
      .format("kafka") \
      .option("kafka.bootstrap.servers", KafkaServer) \
      .option("subscribe", KafkaTopics) \
      .load()
  
  Transaction_DF = df.selectExpr("CAST(value AS STRING)")
  
  decompDF = Transaction_DF.select(zip_extract("value").alias("decompress"))
  filterDF = decomDF.filter(.....) 
  
  query = filterDF .writeStream \
      .option("path", outputpath) \
      .option("checkpointLocation", RawXMLCheckpoint) \
      .start()
  </code></pre>