title: improve speed of pandas to_csv() to write large data to disk
body: <p>I have a large dataframe of size 200 Million rows and 50 Columns. The dataframe contains text as well as numeric data.</p>
  
  <p>I am using pandas to write the dataframe to disk, below is the <code>to_csv()</code> function as i am using,</p>
  
  <pre><code>df.to_csv("test.csv",index=True,header=True,chunksize=100000)
  </code></pre>
  
  <p>I have read multiple similar threads about this but could not find any suitable answer. Below are the points that i have tried,</p>
  
  <ul>
  <li>I have already tried multiple times by varying chunksize, using encoding and compressions but none of them give a considerable performance enhancement.</li>
  <li><p>I have tried writing the files to <code>tmpfs</code> (ram disk) as well, but no improvement, providing steps below for further validation,</p>
  
  <p><code>mount -t tmpfs -o size=50000m tmpfs /tmp-disk/</code></p>
  
  <p>and then</p>
  
  <p><code>df.to_csv("/tmp-disk/test.csv",index=True,header=True,chunksize=100000)</code></p></li>
  </ul>
  
  <p>Also, splitting and starting multiple threads is not an option for me here. And I need to output csv only.</p>