title: PySpark: Counting Word Pairs and Individual Words
body: <p>I am trying to do some basic text analysis using PySpark. </p>
  
  <p>To start this problem, I load a text file containing shakespears sonnets to an RDD. The output should be an RDD containing tuples as shown below: </p>
  
  <pre><code>(word_pair, count_of_word_pair, word_1_count, word_2_count)
  </code></pre>
  
  <p>where word_1 and word_2 are the individual words that make up the word_pair.</p>
  
  <p>So far, I am able to correctly generate one RDD that contains the individual unique words and the count of the number of lines that they occur on and another RDD containing word_pairs (order matters) and counts:    </p>
  
  <pre><code>distFile = sc.textFile("Shakespeare.txt")
  
      tokens = (
          distFile.map(simple_tokenize)
          .flatMap(set)
          .map(lambda x: (x, 1))
          .reduceByKey(lambda x, y: x + y)
      )
  
      token_pairs = (
          distFile.map(simple_tokenize)
          .flatMap(lambda x: itertools.permutations(x, 2))
          .filter(lambda x: x[0] != x[1])
          .map(lambda x: (x, 1))
          .reduceByKey(lambda x, y: x + y)
  
  </code></pre>
  
  <p><strong>Note:</strong> <em>simple_tokenize</em> is a function that sets all characters to lowercase and splits the sentence on each line into a list of words. </p>
  
  <p>Sample outputs below:
  First 10 Elements of tokens:</p>
  
  <pre><code>('shakespeare', 38),
    ('we', 3125),
    ('fairest', 42),
    ('increase', 28),
    ('creatures', 38),
    ('die', 461),
    ('rose', 68),
    ("beauty's", 30),
    ('thereby', 26),
    ('never', 996)]
  </code></pre>
  
  <p>First 10 elements of token_pairs:</p>
  
  <pre><code>
  [('the', 'sonnets', 2),
    ('sonnets', 'the', 2),
    ('by', 'william', 39),
    ('william', 'by', 39),
    ('from', 'desire', 4),
    ('fairest', 'creatures', 1),
    ('fairest', 'we', 1),
    ('fairest', 'increase', 1),
    ('creatures', 'fairest', 1),
    ('creatures', 'we', 4)],
  </code></pre>
  
  <p>I am very stuck in that I do not understand how to combine these different counts in the same RDD (as per the required output). As I understand it, I cannot perform any type of key loop between RDDs inside of a map function (or that I can't do it at all unless I collect one of them). Is this correct?</p>
  
  <p>How can I get the desired output without having to store anything as a local variable in Python?</p>