title: Reading and handling many small CSV-s to concatenate one large Dataframe
body: <p>I have two folders each contains about 8,000 small csv files. One with an aggregated size of around 2GB and another with aggregated size of around 200GB.</p>
  <p>These files are stored like this to better update them in a daily basis. However, when I conduct EDA, I would like them to be assigned to a single variable. For example.</p>
  <pre><code>path = &quot;some random path&quot;
  df = pd.concat([pd.read_csv(f&quot;{path}//{files}&quot;) for files in os.listdir(path)])
  </code></pre>
  <p>It would take much less time for me to read the dataset with 2GB in total size than reading it on the super computer cluster. And it is impossible to read the 200GB dataset on the local machine unless using some sort of scaling Pandas solutions. The situation does not seem to improve on the cluster even using the popular open-source tools like Dask and Modin.</p>
  <p>Is there an effective way that enables to read those csv files effectively with given situation?</p>