title: Is there a way to write RDD rows to HDFS or S3 from inside a map transformation?
body: <p>I'm aware that the typical way of writing RDD or Dataframe rows to HDFS or S3 is by using <code>saveAsTextFile</code> or <code>df.write</code>. However, I would like to figure out how to write individual records from inside a map transformation like this:</p>
  
  <pre><code>myRDD.map(row =&gt; {
    if(row.contains("something")) {
      // write record to HDFS or S3
    }
    row
  }
  </code></pre>
  
  <p>I know that this can be accomplished with the following code,</p>
  
  <pre><code>val newRDD = myRDD.filter(row =&gt; row.contains("something"))
  newRDD.saveAsTextFile("myFile")
  </code></pre>
  
  <p>but I want to continue processing the original <code>myRDD</code> after writing to HDFS and that would require caching <code>myRDD</code> and I am low on memory resources.</p>