title: Saving a >>25T SchemaRDD in Parquet format on S3
body: <p>I have encountered a number of problems when trying to save a very large SchemaRDD as in Parquet format on S3. I have already posted specific questions for those problems, but this is what I really need to do. The code should look something like this</p>
  
  <pre><code>import org.apache.spark._
  val sqlContext = sql.SQLContext(sc)
  val data = sqlContext.jsonFile("s3n://...", 10e-6)
  data.saveAsParquetFile("s3n://...")
  </code></pre>
  
  <p>I run into problems if I have more than about 2000 partitions or if there is partition larger than 5G.
  This puts an upper bound on the maximum size SchemaRDD I can process this way.
  The prctical limit is closer to 1T since partitions sizes vary widely and you only need 1 5G partition to have the process fail.</p>
  
  <p>Questions dealing with the specific problems I have encountered are</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/26321947/multipart-uploads-to-amazon-s3-from-apache-spark">Multipart uploads to Amazon S3 from Apache Spark</a></li>
  <li><a href="https://stackoverflow.com/questions/26320274/error-when-writing-a-repartitioned-schemardd-to-parquet-with-spark-sql">Error when writing a repartitioned SchemaRDD to Parquet with Spark SQL</a></li>
  <li><a href="https://stackoverflow.com/questions/26291165/spark-sql-unable-to-complete-writing-parquet-data-with-a-large-number-of-shards">Spark SQL unable to complete writing Parquet data with a large number of shards</a></li>
  </ul>
  
  <p>This questions is to see if there are any solutions to the main goal that do not necessarily involve solving one the above problems directly.</p>
  
  <hr>
  
  <p>To distill things down there are 2 problems</p>
  
  <ul>
  <li><p>Writing a single shard larger than 5G to S3 fails. AFAIK this a built in limit of <code>s3n://</code> buckets. It should be possible for <code>s3://</code> buckets but does not seem to work from Spark and <code>hadoop distcp</code> from local HDFS can not do it either.</p></li>
  <li><p>Writing the summary file tends to fail once there are 1000s of shards. There seem to be multiple issues with this. Writing directly to S3 produces the error in the linked question above. Writing directly to local HDFS produces an OOM error even on an r3.8xlarge (244G ram) once when there about 5000 shards. This seems to be independent of the actual data volume. The summary file seems essential for efficient querying.</p></li>
  </ul>
  
  <p>Taken together these problems limit Parquet tables on S3 to 25T. In practice it is actually significantly less since shard sizes can vary widely within an RDD and the 5G limit applies to the largest shard.</p>
  
  <p><strong>How can I write a >>25T RDD as Parquet to S3?</strong></p>
  
  <p>I am using Spark-1.1.0.</p>