title: How does one append large amounts of data to a Pandas HDFStore and get a natural unique index?
body: <p>I'm importing large amounts of http logs (80GB+) into a Pandas HDFStore for statistical processing. Even within a single import file I need to batch the content as I load it. My tactic thus far has been to read the parsed lines into a DataFrame then store the DataFrame into the HDFStore. My goal is to have the index key unique for a single key in the DataStore but each DataFrame restarts it's own index value again. I was anticipating HDFStore.append() would have some mechanism to tell it to ignore the DataFrame index values and just keep adding to my HDFStore key's existing index values but cannot seem to find it. How do I import DataFrames and ignore the index values contained therein while having the HDFStore increment its existing index values? Sample code below batches every 10 lines. Naturally the real thing would be larger.</p>
  
  <pre class="lang-py prettyprint-override"><code>if hd_file_name:
          """
          HDF5 output file specified.
          """
  
          hdf_output = pd.HDFStore(hd_file_name, complib='blosc')
          print hdf_output
  
          columns = ['source', 'ip', 'unknown', 'user', 'timestamp', 'http_verb', 'path', 'protocol', 'http_result', 
                     'response_size', 'referrer', 'user_agent', 'response_time']
  
          source_name = str(log_file.name.rsplit('/')[-1])   # HDF5 Tables don't play nice with unicode so explicit str(). :(
  
          batch = []
  
          for count, line in enumerate(log_file,1):
              data = parse_line(line, rejected_output = reject_output)
  
              # Add our source file name to the beginning.
              data.insert(0, source_name )    
              batch.append(data)
  
              if not (count % 10):
                  df = pd.DataFrame( batch, columns = columns )
                  hdf_output.append(KEY_NAME, df)
                  batch = []
  
          if (count % 10):
              df = pd.DataFrame( batch, columns = columns )
              hdf_output.append(KEY_NAME, df)
  </code></pre>