In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training. However, earlier stages of the training pipeline, such as disk I/O and data preprocessing, do not run on accelerators. As accelerators continue to improve, these earlier stages will increasingly become the bottleneck. In this paper, we introduce "data echoing," which reduces the total computation used by earlier pipeline stages and speeds up training whenever computation upstream from accelerators dominates the training time. Data echoing reuses (or "echoes") intermediate outputs from earlier pipeline stages in order to reclaim idle capacity. We investigate the behavior of different data echoing algorithms on various workloads, for various amounts of echoing, and for various batch sizes. We find that in all settings, at least one data echoing algorithm can match the baseline's predictive performance using less upstream computation. We measured a factor of 3.25 decrease in wall-clock time for ResNet-50 on ImageNet when reading training data over a network.
The authors propose a simple method for avoiding bottlenecks during NN training, whereby training examples are utilized multiple times per read. The work focuses on cases where the cost of preparing a minibatch exceeds that of a training step (both a forward pass and, subsequently, a parameter update). Working within said regime, the authors investigate different strategies for 'echoing' examples.
Feedback:
  The proposed method itself is very simple: that's fine. While some cursory analysis of data echoing's theoretical implications would be appreciated, I am fine with practically motivated solutions that address real issues. Simple 'tricks' that are easy to implement and widely applicable are often useful tools. Especially when little theoretical analysis is provided, introducing such a trick requires strong empirical evidence to validate its efficacy. As it stands, failure to provide crucial information forces the reader to suspend disbelief when evaluating the proposed method's impact.
  The authors seem to tiptoe around the issue of the relative cost of prefetching a batch versus that of a combined forward pass and parameter update. The work is predicated upon the assumption that the ratio of said costs $R > 1$, but the authors state that an unspecified subset of their experiments violate this assumption. What's more, real-world values of $R$ are not reported (or even measured!). As per Amdahl's law, $R$ upper bounds the potential benefits for data echoing; hence, failing to report $R$ is more than a little bit concerning. By the same token, the appropriate statistic for various result figures would seemingly be time rather than, e.g., the number of fresh examples read. As a reviewer, I would rather see evidence that data echoing provides modest benefits in realistic scenarios than x3.25 speedup in self-described "contrived" examples. Alternatively, consider providing real-world examples where $R > 1$ to help ground your arguments.
Questions:
  - Why was extensive hyperparameter tuning necessary?
  - Why are most results reported in terms of time/steps to achieve a target value? If nothing else, consider providing the corresponding learning curves (as in Figure 8) as an appendix (incl. means and standard errors).
Nitpicks, Spelling, & Grammar:
  - Metaparameters -> hyperparameters
  - Streamline list at end of introduction:
    """
    In this paper, we demonstrate that data echoing:
        1. reduces the...
    """