A successful application of convolutional architectures is to increase the resolution of single low-resolution images -a image restoration task called super-resolution (SR). Naturally, SR is of value to resource constrained devices like mobile phones, electronic photograph frames and televisions to enhance image quality. However, SR demands perhaps the most extreme amounts of memory and compute operations of any mainstream vision task known today, preventing SR from being deployed to devices that require them. In this paper, we perform a early systematic study of system resource efficiency for SR, within the context of a variety of architectural and low-precision approaches originally developed for discriminative neural networks. We present a rich set of insights, representative SR architectures, and efficiency trade-offs; for example, the prioritization of ways to compress models to reach a specific memory and computation target and techniques to compact SR models so that they are suitable for DSPs and FPGAs. As a result of doing so, we manage to achieve better and comparable performance with previous models in the existing literature, highlighting the practicality of using existing efficiency techniques in SR tasks. Collectively, we believe these results provides the foundation for further research into the little explored area of resource efficiency for SR.
The paper proposes a detailed empirical evaluation of the trade-offs achieved by various convolutional neural networks on the super resolution problem. The paper provides an extensive evaluation of different architectural changes and the trade-off between savings in terms of memory and computational cost and performance, measured in terms of PSNR and SSIM.
This is an empirical paper, thus it does not provide technical contributions. I do think that the insights obtained from such an empirical evaluation could be of interest for practitioners and researchers working on the problem. My main concern is the method only evaluates the trade-offs between model efficiency (in terms of memory and/or computation) and performance measured using metrics that are known to not be well correlated with perceptual quality. Thus it is not obvious to me that the insights obtained in this work would translate to the other case.
It is well known that PSNR favors blurry solutions over perceptually more appealing solutions. This comes from the fact that there is no information in the low resolution image to produce the missing high resolution details. Filling up plausible details in a way that is different from the original image would lead to high PSNR. Models that treat the super resolution problem as a regression task using similarity in pixel space, tend to produce blurry solutions and require very large models to improve the score.  
In recent years, many works have been studying the use of perceptual losses to mitigate this issue or simply treating the super resolution problem as conditional generative modeling.  For instance, models using L2 losses in a perceptually more relevant (or learned) feature spaces [A, B], or including GAN losses [C, D] (to list a few). To my knowledge, these models are the current state of the art in terms of perceptual quality. This has been evaluated empirically via perceptual tests [D].  
This line of work needs to be cited. In my view, the paper needs to provide a detailed justification on why models using these losses are not considered. Would the conclusions drawn on this work transfer to that setting? Furthermore, it would be good to perform perceptual tests to perform this evaluation. It would be good to provide some canonical examples in the appendix.
The overall writing of the paper could be improved. Several sentences are difficult to read, due to typos or the construction of the sentences. The paper evaluates many architectural modifications proposed by other works. It would be good to add an appendix with a small description of what these are. This would make the paper self-contained an easier to read (I had too look up a few of them).
The authors mentioned that they first train models for scaling factor of x2 and then use them for training settings higher magnification. How is this exactly done? Please provide details.
I am curious of weather using some for of distillation techniques would be useful here.
Did you try scaling factors larger than x4? Scaling factors of x2 does not seem very relevant, as simpler methods can achieve already quite competitive results (such as simple interpolation methods)
The authors seem to be citing Zhang et al (2018) as a reference to attention mechanisms. To my knowledge the paper that proposed these mechanisms is [E].
The citation style is not used properly throughout the manuscript. As an example:
“… proposed in StrassenNets Tschannen et al (2017).” Should be “… proposed in StrassenNets (Tschannen et al, 2017).” Or “… proposed in StrassenNets proposed by Tschannen et al (2017).”
[A] Johnson, J. et al. "Perceptual losses for real-time style transfer and super-resolution." ECCV, 2016.
[B] Bruna, J. et al "Super-resolution with deep convolutional sufficient statistics." ICLR 2016.
[C] Ledig, C. et al. "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network." CVPR. Vol. 2. No. 3. 2017.
[D] Sønderby, C. K., et al. "Amortised map inference for image super-resolution." arXiv preprint arXiv:1610.04490(2016).
[E] Bahdanau, D. et al "Neural machine translation by jointly learning to align and translate." arXiv (2014).