title: Pyspark: Read multiple csv files and annotate them with the source
body: <p>We've been running into issues with bulk file ingestion into spark.</p>
  <p>Currently, I'm aware that multiple file ingestion can be done with wildcards</p>
  <pre><code>spark.read.csv(&quot;path/to/file*.csv&quot;)
  </code></pre>
  <p>or via passing a list of paths of interest</p>
  <pre><code>spark.read.csv([&quot;path/to/file1.csv&quot;, &quot;path/to/file2.csv&quot;])
  </code></pre>
  <p>In our situation, we have a large number of files (&gt;100k) with file_name encoded IDs and no id encoded within the table itself. Using either method above acts like a simple union of the files, and doesn't seem to allow for the storage of the filename anywhere within the dataset.</p>
  <p>How would I go about combining all these csvs while maintaining the filename encoded ID.</p>