Using Recurrent Neural Networks (RNNs) in sequence modeling tasks is promising in delivering high-quality results but challenging to meet stringent latency requirements because of the memory-bound execution pattern of RNNs. We propose a big-little dual-module inference to dynamically skip unnecessary memory access and computation to speedup RNN inference. Leveraging the error-resilient feature of nonlinear activation functions used in RNNs, we propose to use a lightweight little module that approximates the original RNN layer, which is referred to as the big module, to compute activations of the insensitive region that are more errorresilient. The expensive memory access and computation of the big module can be reduced as the results are only used in the sensitive region. Our method can reduce the overall memory access by 40% on average and achieve 1.54x to 1.75x speedup on CPU-based server platform with negligible impact on model quality.
In this paper, the authors design a big-little dual-module inference to dynamically skip unnecessary memory access and computation to speedup RNN inference.
It cannot only mitigate the memory-bound problem to speedup RNN inference but also leverage the error resilience of nonlinear activation functions by using the lightweight little module to compute for the insensitive region and using the big module with skipped memory access and computation. They also conduct several experiments to evaluate their approaches.
Strength:
(1)	Well written in general.
(2)	Contributions clearly stated and justified
A couple of minor questions:
(1)	The organization in Section 3 can to be improved. It is better if the author can give a brief overview of their method first and then go into details.
(2)	Some of the technical details necessary for understanding the soundness of the techniques are either missing or are poorly explained. For example, in Section 3
a.	the authors did not mention how to construct the HH module
b.	the authors did not provide detailed information of how to conduct dimension reduction since this will affect the performance
c.	many mathematical notations and equations need to be revised to increase the readability. For instance, there is no information in the paper that explain why the authors design functions in such certain way (such as equation (2) and (3))
d.	the authors did not provide enough detailed information about how to select the quantization methods since there are lots of approaches such as static (uniform) or dynamic quantization, where different methods may have different impacts on the final performance 
e.	the authors mentioned that they have tried both sigmoid and tanh activation function to find the sensitive region. However, they do not provide enough reason to do so, how about using other non-linear activation functions
(3)	The organization in Section 4 can to be improved. It is better if the author can introduce the motivation of each experiment.
(4)	Parameters of the evaluation are unclear or missing. For example, 
a.	what is the data size, what is the dropout, learning rate, how many time stamps for the RNN modules
b.	why the authors only use single-layer LSTM and why they select 750 and 1500 hidden units in the experiments
(5)	While the authors have applied their models on other existing method, they do not provide good discussion of results and such model seems old (released in 2016). It would be great if this approach can also be applied on other newly released models
(6)	Some tables need to be reorganized. For instance, for table 6, there needs some space between the title and the table.
(7)	While the paper has good coverage of the prior work, I do suggest the authors can also cite or discuss some newly designed models (in 2019).