title: Is it possible to get the first n elements of every RDD in Spark Streaming?
body: <p>When using Spark Streaming, is it possible to get the first <code>n</code> elements of every RDD in a DStream? In the real world, my stream consists of a number of geotagged events, and I want to take the 100 (or whatever) which are closest to a given point for further processing, but a simple example which shows what I'm trying to do is something like:</p>
  
  <pre><code>import org.apache.spark.SparkConf
  import org.apache.spark.streaming.dstream.ConstantInputDStream
  import org.apache.spark.streaming.{Seconds, StreamingContext}
  
  object take {
    def main(args: Array[String]) {
  
      val data = 1 to 10
  
      val sparkConf = new SparkConf().setAppName("Take");
      val streamingContext = new StreamingContext(sparkConf, Seconds(1))
  
      val rdd = streamingContext.sparkContext.makeRDD(data)
      val stream = new ConstantInputDStream(streamingContext, rdd)
  
      // In the real world, do a bunch of stuff which results in an ordered RDD
  
      // This obviously doesn't work
      // val filtered = stream.transform { _.take(5) }
  
      // In the real world, do some more processing on the DStream
  
      stream.print()
  
      streamingContext.start()
      streamingContext.awaitTermination()
    }
  }
  </code></pre>
  
  <p>I understand I could pull the top <code>n</code> results back to the driver fairly easily, but that isn't something I want to do in this case as I need to do further processing on the RDD after having filtered it down.</p>