title: How to directly write to HDFS file rather than using saveAsTextFile in python spark?
body: <p>I have reached to some point where I have got the result of a <code>groupByKey</code> operation. Now I want to write each <code>(key, value)</code> pair into different files using <code>k</code> as their names and <code>v</code> as their content.</p>
  
  <p>Firstly, I tried to collect those results in the driver program in order to use <code>open()</code> and write to local files, but it failed because of a buffer overflow as the size of the results is very large.</p>
  
  <p>I then tried to filter those <code>(key, value)</code> pair to make new RDD in order to use <code>saveAsTextFile</code> for each new RDD, but it seems to become too slow for the network communications. </p>
  
  <p>Now I think maybe it should be faster if I can use a <code>foreach</code> operation over the <code>groupByKey</code> RDD and write to HDFS files directly in each iteration, but I have no idea which function to refer to in the python API. Could anyone show me some examples or tell me another way to accomplish my goal in pyspark?</p>
  
  <p>There is a similar question <a href="https://stackoverflow.com/questions/23995040/write-to-multiple-outputs-by-key-spark-one-spark-job">here</a> on SO, however that solution is in scala while I want to solve it in python, because I found no way to communicate directly with HDFS in pyspark.</p>