title: Pandas Dataframe: to_dict() poor performance
body: <p>I work with apis that return large pandas dataframes. I'm not aware of a fast way to iterate through the dataframe directly so I cast to a dictionary with <code>to_dict()</code>.</p>
  
  <p>After my data is in dictionary form, the performance is fine. However, the <code>to_dict()</code> operation tends to be a performance bottleneck. </p>
  
  <p>I often group columns of the dataframe together to form multi-index and use the 'index' orientation for <code>to_dict()</code>. Not sure if the large multi-index drives the poor performance. </p>
  
  <p>Is there a faster way to cast a pandas dataframe? Maybe there is a better way to iterate directly over the dataframe without any cast? Not sure if there is a way I could apply vectorization.</p>
  
  <p>Below I give sample code which mimics the issue with timings:</p>
  
  <pre><code>import pandas as pd
  import random as rd
  import time
  
  #Given a dataframe from api (model as random numbers)
  df_columns = ['A','B','C','D','F','G','H','I']
  dict_origin = {col:[rd.randint(0,10) for x in range(0,1000)] for col in df_columns}
  dict_origin = pd.DataFrame(dict_origin)
  
  #Transform to pivot table
  t0 = time.time()
  df_pivot = pd.pivot_table(dict_origin,values=df_columns[-3:],index=df_columns[:-3])
  t1 = time.time()
  print('Pivot Construction takes: ' + str(t1-t0))
  
  #Iterate over all elements in pivot table
  t0 = time.time()
  for column in df_pivot.columns:
      for row in df_pivot[column].index:
          test = df_pivot[column].loc[row]
  t1 = time.time()
  print('Dataframe iteration takes: ' + str(t1-t0))
  
  
  #Iteration over dataframe too slow. Cast to dictionary (bottleneck)
  t0 = time.time()
  df_pivot = df_pivot.to_dict('index')
  t1 = time.time()
  print('Cast to dictionary takes: ' + str(t1-t0))
  
  #Iteration over dictionary is much faster
  t0 = time.time()
  for row in df_pivot.keys():
      for column in df_pivot[row]:
          test = df_pivot[row][column]
  t1 = time.time()
  print('Iteration over dictionary takes: ' + str(t1-t0))
  </code></pre>
  
  <p>Thank you!</p>