Thanks for sharing your story!<p>Let me comment on file formats as someone familiar with both netCDF and deep learning.<p>I agree that netCDF is a sane binary file format for this application. It&#x27;s designed for efficient serialization of large arrays of numbers. One downside is that netCDF does not support streaming without writing the data to intermediate files on disk.<p>Keep in mind that netCDF v4 is itself just a thin wrapper around HDF5. Given that your input format is basically a custom file format written in netCDF, I would have just used HDF5 directly. The API is about as convenient, and this would skip one layer of indirection.<p>The native file format for TensorFlow is its own custom TFRecords file format, but it also supports a number of other file formats. TFRecords is <i>much</i> simpler technology than NetCDF&#x2F;HDF5. It&#x27;s basically just a bunch of serialized protocol buffers [1]. About all you can do with a TFRecords file is pull out examples -- it doesn&#x27;t support the fancy multi-dimensional indexing or hierarchical structure of netCDF&#x2F;HDF5. But that&#x27;s also most of what you need for building machine learning models, and it&#x27;s quite straightforward to read&#x2F;write them in a streaming fashion, which makes it a natural fit for technologies like map-reduce.<p>[1] <a href="https:&#x2F;&#x2F;www.tensorflow.org&#x2F;versions&#x2F;r0.8&#x2F;api_docs&#x2F;python&#x2F;python_io.html#tfrecords-format-details" rel="nofollow">https:&#x2F;&#x2F;www.tensorflow.org&#x2F;versions&#x2F;r0.8&#x2F;api_docs&#x2F;python&#x2F;pyt...</a>