Deep learning training accesses vast amounts of data at high velocity, posing challenges for datasets retrieved over commodity networks and storage devices. We introduce a way to dynamically reduce the overhead of fetching and transporting training data with a method we term Progressive Compressed Records (PCRs). PCRs deviate from previous formats by using progressive compression to convert a single dataset into multiple datasets of increasing fidelity-all without adding to the total dataset size. Empirically, we implement PCRs and evaluate them on a wide range of datasets: ImageNet, HAM10000, Stanford Cars, and CelebA-HQ. Our results show that different tasks can tolerate different levels of compression. PCRs use an on-disk layout that enables applications to efficiently and dynamically access appropriate levels of compression at runtime. In turn, we demonstrate that PCRs can seamlessly enable a 2× speedup in training time on average over baseline formats.
Summary: This paper introduces a new storage format for image datasets for machine learning training. The core idea is to use progressive JPEG to create sequential scans of the input image, from lower resolution to higher resolution. The authors found that on some datasets, using half of the scans is already enough to reach similar accuracy but speeded up the convergence by a factor of 2.
Detailed feedbacks:
-	The paper presents a simple idea that directly uses the nature of JPEG compression. The paper shows that it can work well and can be potentially integrated into real machine learning dataset storage applications.
-	Related work section is thorough.
-	The experiments are limited to image classifications, and some of the datasets are subsampled (e.g. ImageNet and CelebA). This may not well represent real machine learning tasks, and practitioners may be unsure about the reliability of the compression. The “Cars” dataset contains fine-grained classification, in which the proposed method is
-	Figure 1 is not very clear what is the key advantage of the proposed method, and what are the different mechanisms.
-	Alternatively, one can subsample the pixels and store incremental subsets of those pixels. It would be good if the paper can discuss about this baseline.
-	The data storage format is only loosely related to the main goal of the paper, which is to show that network can still train very well and even faster when receiving partial input data. Once they figured out the number of scans needed for this application, they don’t necessarily need to keep a full lossless version and can just go for a lossy version. In other words, the experiment section can be replaced by any other lossy compression by varying the compression ratio.
-	In my opinion, there could be two reasons for faster convergence. 1) lowered image quality makes the data easier to learn and 2) the smaller data size allows faster reading of data from disk. The paper only shows wall-clock speed-up, but it is unclear which factor is bigger. 2) can be potentially addressed by faster disk reading such as SSD or in-memory datasets. One of the motivations is to help parallel training of dataset and it is also mentioned how non-random sampling of data can hurt training performance. It would be good to showcase how the proposed method can help in those parallel training settings. 
Conclusion: This paper presents a simple and effective idea and can be potentially beneficial. However, my main concern is whether the experiments can be representative enough for large scale experiments (e.g. using non-subsampled ImageNet dataset with parallel training using SSD storage). Therefore, my overall rating is weak accept.