title: Optimal file size and parquet block size
body: <p>I have around 100 GB of data per day which I write to S3 using Spark. The write format is parquet. The application which writes this run Spark 2.3</p>
  
  <p>The 100 GB data is further partitioned, where the largest partition is 30 GB. For this case, let's just consider that 30 GB partition.</p>
  
  <p>We are planning to migrate this whole data and rewrite to S3, in Spark 2.4. Initially we didn't decide on file size and block size when writing to S3. Now that we are going to rewrite everything, we want to take into consideration the optimal file size and parquet block size.</p>
  
  <ol>
  <li>What is the optimal file size to write to S3 in parquet ?</li>
  <li>Can we write 1 file with 30 GB size and parquet block size as 512 MB ? How will reading work in this case ?</li>
  <li>Same as #2 but parquet block size as 1 GB ?</li>
  </ol>