title: Efficiently storing 2D heat maps for loading at runtime in Keras
body: <p>I would like to train an autoencoder (with convolutional blocks) on a massive amount of heat maps using Keras. The heat maps are 2D and of size 1000 x 1000 x 1 (one color channel). The heat maps aggregate usage statistics of several users (for example web browser usage). For each user I create around 1000 heat maps and in total I have 100 users, so a total amount of 100'000 heat maps. The heat maps are relatively sparse (i.e. a lot of zero entries).</p>
  <p>My problem now is how to efficiently store these heat maps. The heat maps cannot be held in memory all at once. I could store the heat maps as png or jpg files but the problem is that it takes a lot of time store 100'000 heat maps as jpg or png and I don't know if it is then very efficient to read these files. Another possibility would be to store all heat maps for each user in one file such as a hdf. The problem is that I would like to split the data into a training, validation and test set and load the data at runtime. When I store each heat map separately (e.g. as a jpg) this is possible using a <code>DataGenerator</code> of Keras. But when I have one file of heat maps for each user, I don't know how I can do the split in training, validation and test to load the heat maps at runtime (in cases where the split is not per user).</p>
  <p>What is the best way to store such heat maps?</p>