title: Pyspark dataframe split alphabetically and write to S3
body: <p>I am trying to split a huge XML file into small XML files using pyspark. I need the data to be written into buckets alphabetically.</p>
  <p>Suppose if the name starts with <code>a</code> then it would be written to an s3 bucket <code>s3://bucket_name/a</code>. If there is no name that starts with <code>b</code> it  should still create a folder with name b in the same bucket, that is <code>s3://bucket_name/b</code></p>
  <p>So far the code I have is</p>
  <pre><code>charater = [&quot;a&quot;, &quot;b&quot;, &quot;c&quot;, &quot;d&quot;, &quot;e&quot;, &quot;f&quot;, &quot;g&quot;, &quot;h&quot;, &quot;i&quot;, &quot;j&quot;, &quot;k&quot;, &quot;l&quot;, &quot;m&quot;, &quot;n&quot;, &quot;o&quot;, &quot;p&quot;, &quot;q&quot;, &quot;r&quot;, &quot;s&quot;, &quot;t&quot;, &quot;u&quot;, &quot;v&quot;, &quot;w&quot;, &quot;x&quot;, &quot;y&quot;, &quot;z&quot;]
  
  for c in charater:
      df1 = df.filter(lower(trim(col('name')).substr(1, 1)) == c)
      df1.write\
       .format(&quot;com.databricks.spark.xml&quot;)\
       .option(&quot;maxRecordsPerFile&quot;, 800)\
       .option(&quot;rootTag&quot;,&quot;source&quot;)\
       .option(&quot;rowTag&quot;, &quot;employees&quot;)\
       .mode(&quot;overwrite&quot;)\
       .save(f's3://split-files/{c}')
  
  </code></pre>
  <p>But this code takes a very long time to finish. Is there a better way to do this using data frames?</p>
  <p>Thanks in advance.</p>