title: Writing Data to h5py on SSD disk appears slow: What can I do to speed it up
body: <p>I'm trying to write data to a h5py dataset but using a high memory 12 core GCE instance to write to an SSD disk but it runs for 13 hours with no end in sight. 
  I'm running Jupyter Notebook on the GCE instance to unpickle a large number of small files (stored on a 2nd non-ssd disk) before adding them to a h5py dataset in a file stored on the ssd disk</p>
  
  <ul>
  <li>Max shape=<code>(29914, 251328)</code> </li>
  <li>Chunks = <code>(59, 982)</code></li>
  <li>compression = <code>gzip</code></li>
  <li>dtype = <code>float64</code></li>
  </ul>
  
  <p>My code is listed below</p>
  
  <pre><code>#Get a sample
  minsample = 13300
  sampleWithOutReplacement = random.sample(ListOfPickles,minsample)
  
  print(h5pyfile)
  with h5py.File(h5pyfile, 'r+') as hf:
      GroupToStore = hf.get('group')
      DatasetToStore = GroupToStore.get('ds1')
      #Unpickle the contents and add in th h5py file                
      for idx,files in enumerate(sampleWithOutReplacement):
          #Sample the minimum number of examples
          time FilePath = os.path.join(SourceOfPickles,files)
          #Use this method to auto close the file
          with open(FilePath,"rb") as f:
              %time DatasetToStore[idx:] = pickle.load(f)
              #print("Processing file ",idx)
  
  print("File Closed")
  </code></pre>
  
  <p>The h5py File on disk seems to increase 1.4GB each dataset I populate using the code above and below is my code to create the dataset in the h5py file</p>
  
  <pre><code>group.create_dataset(labels, dtype='float64',shape= (maxSize, 251328),maxshape=(maxSize,251328),compression="gzip")
  </code></pre>
  
  <p>What improvements can I make to either my configuration or my code or both to reduce the time needed to populate the h5py file? </p>
  
  <p><strong>Update 1</strong>
   I added some magic to the jupyter notebook to time the process, I'd welcome any advice on speeding up the loading into the datastore which was reported as taking <strong>8hrs</strong></p>
  
  <pre><code>CPU times: user 0 ns, sys: 0 ns, total: 0 ns
  Wall time: 14.1 µs
  CPU times: user 8h 4min 11s, sys: 1min 18s, total: 8h 5min 30s
  Wall time: 8h 5min 29s      
  </code></pre>