title: Spark: Fastest way to look up an element in an RDD
body: <p>I have a custom class E which has, among others, a field <code>word</code>. I have a large <code>es: RDD[E]</code> with several 100000 elements and a <code>doc: Seq[String]</code> with typically a few hundred entries. In <code>es</code>, every element's <code>word</code> field value is unique.</p>
  
  <p>My task is to look up the element in <code>es</code> for each of the strings in <code>doc</code>. It is however not guaranteed that such an element exists. My naive Scala/Spark implementation is thus:</p>
  
  <pre><code>def word2E(words: Seq[String]): Seq[E] = {
    words.map(lookupWord(_, es))
      .filter(_.isDefined)
      .map(_.get)
  }
  </code></pre>
  
  <p>The method <code>lookupWord()</code> is defined as follows:</p>
  
  <pre><code>def lookupWord(w: String, es: RDD[E]): Option[E] = {
    val lookup = es.filter(_.word.equals(w))
  
    if (lookup.isEmpty) None
    else Some(lookup.first)
  }
  </code></pre>
  
  <p>When I look at the Spark stages overview, it seems like <code>lookupWord()</code> is a bottleneck. In particular, the <code>isEmpty()</code> calls in <code>lookupWord</code> take relatively long (up to 2s) in some cases.</p>
  
  <p>I have already persisted the <code>es</code> RDD. Is there any other leverage for optimizing such a task or is this just as good as it gets when operating on such a dataset?</p>
  
  <p>I have noticed the <code>lookup()</code> method in <code>PairRDDFunctions</code> and considered to construct a PairRDD in which the <code>word</code> field would serve as the key. Might that help?
  Drawing any conclusions experimentally here is quite hard because there are so many factors involved.</p>