HitNet: Hybrid Ternary Recurrent Neural Network
Quantization is a promising technique to reduce the model size, memory footprint, and computational cost of neural networks for the employment on embedded devices with limited resources. Although quantization has achieved impressive success in convolutional neural networks (CNNs), it still suffers from large accuracy degradation on recurrent neural networks (RNNs), especially in the extremely lowbit cases. In this paper, we first investigate the accuracy degradation of RNNs under different quantization schemes and visualize the distribution of tensor values in the full precision models. Our observation reveals that due to the different distributions of weights and activations, different quantization methods should be used for each part. Accordingly, we propose HitNet, a hybrid ternary RNN, which bridges the accuracy gap between the full precision model and the quantized model with ternary weights and activations. In HitNet, we develop a hybrid quantization method to quantize weights and activations. Moreover, we introduce a sloping factor into the activation functions to address the error-sensitive problem, further closing the mentioned accuracy gap. We test our method on typical RNN models, such as Long-Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). Overall, HitNet can quantize RNN models into ternary values of {-1, 0, 1} and significantly outperform the state-of-the-art methods towards extremely quantized RNNs. Specifically, we improve the perplexity per word (PPW) of a ternary LSTM on Penn Tree Bank (PTB) corpus from 126 to 110.3 and a ternary GRU from 142 to 113.5.
This paper introduces a variation on standard RNN formulations named HitNet. HitNet improves the SOTA of quantization performance for standard RNNs such as LSTM/GRU. Quantization has the benefit of allowing RNNs to operate with vastly reduced memory and computation requirements, but generally introduces a heavy penalty in terms of the relevant loss function (in this case, perplexity).  The authors note that weights and activations in for LSTMs have hugely different distributions (approximately zero mean Gaussian for weights, whereas output activations are within [0,1] with most of the mass being at the extremes of the range). Quantisation for only one of weights or activations, whilst computing the other at full precision, produces a modest decrease in performance, but quantising both at once introduces a heavy penalty. Taking three standard quantization methods (Uniform Quanttization, Threshold Ternary Quantization, and Bernoulli Ternary Quantization) the authors make a compelling case that TTQ is preferred for quantizing weights, whereas BTQ is better for activations, due to the different distributions above. UQ is shown to be a poor proxy for both types of value, even though it has 4 possible values rather than 3.  After the analysis, the authors introduce slope adjustable versions of standard activation function sigmoid and tanh. The sigmoid version was present in previous RBM work cited, whereas the tanh version is (to my knowledge) novel. The temperature parameter lambda controls the slope of the activation, and the authors demonstrate that sweeping this for sigmoid produces more polarised output distributions for an LSTM. By using lambda < 1, the output activations are closer to bipolar, and as such quantization produces less of a performance penalty. The authors choose a range for lambda by sweeping on both a full precision and a quantised model, finding a range of values which performs well for both. HitNet is comprised of the modified activations functions at the relevant places inside LSTM/GRU, as well as BTQ for activations and TTQ for weights.  The authors evaluate on PTB, Wiki-2 and Text8. Significant improvements in perplexity are recorded compared to many other methods of quantizing models for PTB. For the larger datasets, competing benchmarks are not available, but HitNet seems to perform relatively close to the full precision models.  Overall I find this paper very strong. The motivation is clear, the methodology is simply explained and seems easily reproducable. All experiments are described with sufficient hyperparameters, and the improvements over SOTA are significant. I recommend accepting to NIPS.  Minor points:  L120: "The recurrent timestep is set to 35" - I believe this refers to "sequence length" which I would find clearer.  L139: I find the set notation difficult to understand here - should this be "the set of all i, such that |x_i| > \theta"?  A diagram of the modified sigmoid and tanh activation functions, for a range of lambdas, would be an easily added element to aid comprehension.