title: Merging 1300 data frames into a single frame column-wise becomes really slow
body: <p>This is a variation of an earlier question I posted, <a href="https://stackoverflow.com/questions/64388874/merging-1300-data-frames-into-a-single-frame-becomes-really-slow">Merging 1300 data frames into a single frame becomes really slow</a></p>
  <p>This time I am trying to merge the frames column wise not below one another.</p>
  <p>I have 1300 csv files in a directory.</p>
  <p>Each file has a date in the first column, followed by daily data for the last 20-30 years which spans another 8 columns.</p>
  <p>So like this,
  Data1.csv</p>
  <pre><code>Date source1 source2 source3 source4 source5 source6 source 7 source 8
  </code></pre>
  <p>I have 1300 uniquely named files each having uniquely named columns.</p>
  <p>I am trying to merge all of these into one dataframe using pandas column-wise, not appending below but merging left and right like this</p>
  <pre><code>import pandas as pd 
  frame = pd.read_csv(file_path,index_col=0) #this is the location of the first file
  
  
  for filename in os.listdir(filepath): #filepath has the rest of the files
      file_path = os.path.join(filepath, filename)
      df = pd.read_csv(file_path,index_col=0)
      df = df.groupby(['Date']).first()
      df = df.add_prefix(f&quot;{filename}-&quot;)
      frame = pd.merge(frame, df, how='outer', left_index=True, right_index=True)
      length-=1
  
  </code></pre>
  <p>But around the 300th file my code really slows down...</p>
  <p>I am merging these files column wise.</p>
  <p>My goal is actually to have a massive dataframe which is 1+(1300x8) by number of dates for 30 years.</p>
  <p>Is there a way to speed this up before my computer runs out of memory.</p>