title: Append Filename to RDD
body: <p>I have a folder containing my data files. Each file has a size of about 1 GB.</p>
  
  <p>What I need is the filename inside the RDD.</p>
  
  <p>Following does not work as expected:</p>
  
  <pre><code>import glob
  rdds = []
  for filename in glob.iglob('/data/*'):
      rdd = sc.textFile(filename).map(lambda row: (filename, row))
      rdds.append(rdd)
  
  allData = sc.union(rdds)
  </code></pre>
  
  <p>using this, <code>filename</code> is always the filename of the last read file</p>
  
  <p>What I also tried:</p>
  
  <pre><code>import glob
  rdds = []
  for filename in glob.iglob('/data/*'):
      def f(name=filename):
          return name
      rdd = sc.textFile(filename).map(lambda row: (f(), row))
      rdds.append(rdd)
  
  allData = sc.union(rdds)
  </code></pre>
  
  <p>But that produces the error: <code>Broadcast can only be serialized in driver</code></p>
  
  <p><code>sc.wholeTextFile()</code> is not an option because the single files are to big.</p>