Neural language models (NLMs) exist in an accuracy-efficiency tradeoff space where better perplexity typically comes at the cost of greater computation complexity. In a software keyboard application on mobile devices, this translates into higher power consumption and shorter battery life. This paper represents the first attempt, to our knowledge, in exploring accuracy-efficiency tradeoffs for NLMs. Building on quasi-recurrent neural networks (QRNNs), we apply pruning techniques to provide a "knob" to select different operating points. In addition, we propose a simple technique to recover some perplexity using a negligible amount of memory. Our empirical evaluations consider both perplexity as well as energy consumption on a Raspberry Pi, where we demonstrate which methods provide the best perplexity-power consumption operating point. At one operating point, one of the techniques is able to provide energy savings of 40% over the state of the art with only a 17% relative increase in perplexity.
In this paper, the authors investigate the accuracy-efficiency tradeoff for neural language models. In particular, they explore how different compression strategies impact the accuracy (and flops), and more interestingly, also how it impacts the power use for a RaspberryPi. The authors consider the QRNNs and SRUs for this purpose and use standard datasets for their analysis. I am torn about this paper. On one hand, I feel that the analysis is interesting, thoughtful and detailed; the power usage statistics bring a different perspective to the compression community. The section on inference time pruning was especially interesting to read. On the other hand however, there is limited novelty in the setup. The authors use standard, well known, compression algorithms on common neural language modeling architectures and datasets and use out-of-the-box tools for their ultimate analysis. Further, the paper needs additional work before it can be accepted in my opinion. I detail my arguments below:
- The authors begin by discussing SwiftKey and similar apps but I'm not sure if its clear that they use neural language modeling as the backend. Do the authors have a source to validate this claim?
- Knowledge distillation is another algorithm that has been found to be quite competitive in compressing models into smaller versions of themselves. Have the authors experimented with that? 
- The R@3 is an good metric but I suggest that the authors look at mean reciprocal rank (MRR) instead. This removes the arbitrary-ness of "3" while ensuring that the metric of interest is the accuracy and not probability of being correct (perplexity). 
- Can you comment on the sensitivity of the results to the RPi frameworks? For instance, the RPi deployment tools, architecture, and variance in the predictions? 
- Along the same line, I'm curious how generalizable the RPi results are for other computing architectures. For those of us who are not experts on hardware, it would be nice to read about whether similar tradeoffs will exist in other architectures such as mobile phones, GPUs or CPUs. 
- Could the authors add some meta-analysis about the results? If the perplexity goes up as a consequence of compression, what kinds of tokens it that localized to? Is it primarily rare words that the model is less confident about, or are the probabilities for most words getting skewed?
- Finally, I feel that such an exploration will catch on only if the tools are open-sourced and made easy to replicate/use. If there were a blog or article summarizing the steps needed to replicate the power measurement (including sources from where to buy the necessary hardware), more people would be inclined on adding such an analysis to future neural language modeling work. 
I am willing to revisit my rating, as necessary, once I read through the rebuttal. 
UPDATE:
After reading the rebuttal, I am increasing my score to 6. The authors alleviated some of my concerns but my major concerns about their novelty and the impact of their results remains.