title: Write spark dataframe to single parquet file
body: <p>I am trying to do something very simple and I'm having some very stupid struggles. I think it must have to do with a fundamental misunderstanding of what spark is doing. I would greatly appreciate any help or explanation.</p>
  
  <p>I have a very large (~3 TB, ~300MM rows, 25k partitions) table, saved as parquet in s3, and I would like to give someone a tiny sample of it as a single parquet file. Unfortunately, this is taking forever to finish and I don't understand why. I have tried the following:</p>
  
  <pre><code>tiny = spark.sql("SELECT * FROM db.big_table LIMIT 500")
  tiny.coalesce(1).write.saveAsTable("db.tiny_table")
  </code></pre>
  
  <p>and then when that didn't work I tried this, which I <em>thought</em> should be the same, but I wasn't sure. (I added the <code>print</code>'s in an effort to debug.)</p>
  
  <pre><code>tiny = spark.table("db.big_table").limit(500).coalesce(1)
  print(tiny.count())
  print(tiny.show(10))
  tiny.write.saveAsTable("db.tiny_table")
  </code></pre>
  
  <p>When I watch the Yarn UI, <em>both</em> print statements <em>and</em> the <code>write</code> are using 25k mappers. The <code>count</code> took 3 mins, the <code>show</code> took 25 mins, and the <code>write</code> took ~40 mins, although it finally <em>did</em> write the single file table I was looking for.</p>
  
  <p>It seems to me like the first line should take the top 500 rows and coalesce them to a single partition, and then the other lines should happen extremely fast (on a single mapper/reducer). Can anyone see what I'm doing wrong here? I've been told maybe I should use <code>sample</code> instead of <code>limit</code> but as I understand it <code>limit</code> should be much faster. Is that right?</p>
  
  <p>Thanks in advance for any thoughts!</p>