title: Appending map functions to PySpark RDD inside for loop
body: <p>Could someone please help me understand the behaviour of appending map functions to an RDD in a python for loop?</p>
  
  <p>For the following code:</p>
  
  <pre><code>rdd = spark.sparkContext.parallelize([[1], [2], [3]])
  
  def appender(l, i):
      return l + [i]
  
  for i in range(3):
      rdd = rdd.map(lambda x: appender(x, i))
  
  rdd.collect()
  </code></pre>
  
  <p>I get the output:</p>
  
  <pre><code>[[1, 2, 2, 2], [2, 2, 2, 2], [3, 2, 2, 2]]
  </code></pre>
  
  <p>Whereas with the following code:</p>
  
  <pre><code>rdd = spark.sparkContext.parallelize([[1], [2], [3]])
  
  def appender(l, i):
      return l + [i]
  
  rdd = rdd.map(lambda x: appender(x, 1))
  rdd = rdd.map(lambda x: appender(x, 2))
  rdd = rdd.map(lambda x: appender(x, 3))
  
  rdd.collect()
  </code></pre>
  
  <p>I get the expected output:</p>
  
  <pre><code>[[1, 1, 2, 3], [2, 1, 2, 3], [3, 1, 2, 3]]
  </code></pre>
  
  <p>I imagine this has something to do with the closure that is passed to the PySpark  compiler, but I can't find any documentation about this...</p>