title: spark - how to look up the keys and values of a (Java)PairRDD inside another RDD's transformation
body: <p>I have a PairRDD <code>rdd1</code> with Integer keys and Integer[] values. </p>
  
  <p>I also have another PairRDD <code>rdd2</code> with Integer keys and Double values.</p>
  
  <p>Each Integer in the key AND the value of <code>rdd1</code> also exists in <code>rdd2</code> as a key. </p>
  
  <p>I want for each pair <code>(x, [y1,y2,...,yn])</code> in <code>rdd1</code> to get the double value of the <code>x</code> and all the double values of each Integer <code>y1</code>, <code>y2</code>, ..., <code>yn</code>. </p>
  
  <p>I tried collecting <code>rdd2</code> as a <code>Map&lt;Integer,Double&gt;</code> (<code>map2</code>), but it does not fit in memory and I get OOM errors. I also tried joining the rdds, but I could not figure out how to join both the keys and the values. Using <code>rdd2</code>'s <code>lookup()</code> method inside <code>rdd1</code> is not allowed.</p>
  
  <p>The pseudocode of what I want is the following:</p>
  
  <pre><code>map each (int x, int[] y) in rdd1 to:
        (x, map2.get(x) + sum(map2.get(yi)))
  </code></pre>
  
  <p>for each <code>yi</code> in <code>y</code>.</p>
  
  <p>I use Java, but I guess the same problem holds in both Java and Scala.</p>