Faster Neural Networks Straight from JPEG
The simple, elegant approach of training convolutional neural networks (CNNs) directly from RGB pixels has enjoyed overwhelming empirical success. But could more performance be squeezed out of networks by using different input representations? In this paper we propose and explore a simple idea: train CNNs directly on the blockwise discrete cosine transform (DCT) coefficients computed and available in the middle of the JPEG codec. Intuitively, when processing JPEG images using CNNs, it seems unnecessary to decompress a blockwise frequency representation to an expanded pixel representation, shuffle it from CPU to GPU, and then process it with a CNN that will learn something similar to a transform back to frequency representation in its first layers. Why not skip both steps and feed the frequency domain into the network directly? In this paper, we modify libjpeg to produce DCT coefficients directly, modify a ResNet-50 network to accommodate the differently sized and strided input, and evaluate performance on ImageNet. We find networks that are both faster and more accurate, as well as networks with about the same accuracy but 1.77x faster than ResNet-50.
My main concerns were responded to in the the rebuttal (separation between IO/CPU/GPU gains), TFLOP measurements and discussion of related works.  I was happy to see that their model (Late-Concat-RFA-Thinner) is still faster than ResNet 50 (approx 680/475= 43% gains in Fig. 1. c) rebuttal instead of 365/205= 78% in Fig. 4. paper) when feeding the models with zeros disregarding JPEG decoding (CPU time) and IO costs. This is a pessimistic estimate given that the ResNet 50 RGB needs to also do the inverse DCT to go from DCT coefficients to RGB domain.  However, I was a bit surprised to see such a big disconnect between the timing numbers and the TFLOP measurements (Fig. 1. b vs Fig 1. c rebuttal). While I trust that the authors timed the models fairly and thus I do not doubt the results, I think this would be worth more investigation.  For the related works, the authors did a good discussion of them in the rebuttal, but I find it strange that we had to ask for this. (This is something that should obviously be in the initial version of the paper!).  Overall however, accounting for everything, this sentence in the rebuttal resonated with me: "We suggest that without our paper, anyone wanting to run modern deep networks on JPEG representations would first have to re-discover or re-engineer all these critical details that comprise the majority of our paper."  Thus I'm keeping my rating to accept the paper.  ================================  The paper presents a detailed study on a simple but old idea: given that most images are stored as JPEGs, and that the JPEG decoder is nothing but a linear transformation over 8x8 blocks, can we not avoid the JPEG decoding step? There are prior works that have explored similar ideas (which should be properly cited and discussed!), like: * On using CNN with DCT based Image Data Matej Ulicny & Rozenn Dahyot, IMVIP 2017 * Using Compression to Speed Up Image Classification in Artificial Neural Networks Dan Fu, Gabriel Guimaraes, 2016 * Towards Image Understanding from Deep Compression Without Decoding, Torfason et al. ICLR 2018 * High speed deep networks based on Discrete Cosine Transformation, Zou et al, ICIP 2014.  However, aside from Torfason et al. (which do not focus on JPEG but deep compression), these papers are limited in the setting (not going far beyond MNIST classificatoin). In contrast, this paper thoroughly studies it in a large scale setting, showing practical gains for ImageNet classification, exploring the architecture space, e.t.c.  However, I still have some concerns about the paper: * Related works needs more discussion * FPS measurements would be well complemented with TFLOP numbers  Another concern I have is the following: are we seeing implementation speedups or computational speedups? The JPEG decoder is decoding on CPU, whereas the the network is on the GPU. By feeding DCT coefficients to the network, you reduce the CPU workload, so are we just seeing speedups from this workload being moved over to the GPU? The proper way to compare with ResNet-50 RGB would be to use the modified libjpeg created by the authors, and then implement the DCT - > RGB decoding step with a fixed 8x8 deconvolution (carefully checking that it gives the same output as standard JPEG decoder). This should speed up the ResNet50 RGB model, no?  Overall the paper is well written and easy to follow and presents speedup on a standard classification pipeline that are significant.