LEARNING RECURRENT BINARY/TERNARY WEIGHTS
Recurrent neural networks (RNNs) have shown excellent performance in processing sequence data. However, they are both complex and memory intensive due to their recursive nature. These limitations make RNNs difficult to embed on mobile devices requiring real-time processes with limited hardware resources. To address the above issues, we introduce a method that can learn binary and ternary weights during the training phase to facilitate hardware implementations of RNNs. As a result, using this approach replaces all multiply-accumulate operations by simple accumulations, bringing significant benefits to custom hardware in terms of silicon area and power consumption. On the software side, we evaluate the performance (in terms of accuracy) of our method using long short-term memories (LSTMs) and gated recurrent units (GRUs) on various sequential models including sequence classification and language modeling. We demonstrate that our method achieves competitive results on the aforementioned tasks while using binary/ternary weights during the runtime. On the hardware side, we present custom hardware for accelerating the recurrent computations of LSTMs with binary/ternary weights. Ultimately, we show that LSTMs with binary/ternary weights can achieve up to 12× memory saving and 10× inference speedup compared to the full-precision hardware implementation design.
* Summary
This paper proposes batch normalization for learning RNNs with binary or ternary weights instead of full-precision weights. Experiments are carried out on character-level and word-level language modeling, as well as sequential MNIST and question answering.
* Strengths
- I liked the variety of tasks used evaluations (sequential MNIST, language modeling, question answering).
- Encouraging results on specialized hardware implementation.
* Weaknesses
- Using batch normalization on existing binarization/ternarization techniques is a bit of an incremental contribution.
- All test perplexities for word-level language models in table 3 underperform compared to current vanilla LSTMs for that task (see Table 4 in https://arxiv.org/pdf/1707.05589.pdf), suggesting that the baseline LSTM used in this paper is not strong enough.
- Results on question answering are not convincing -- BinaryConnect has the same size while achieving substantially higher accuracy (94.66% vs 40.78%). This is nowhere discussed and the paper's major claims "binaryconnect method fails" and "our method [...] outperforms all the existing quantization methods" seem unfounded (Section 5.5).
- In the introduction, I am lacking a distinction between improvements w.r.t. training vs inference time. As far as I understand, quantization methods only help at reducing memory footprint or computation time during inference/test but not during training. This should be clarified.
- In the introduction on page 2 is argued that the proposed method "eliminates the need for multiplications" -- I do not see how this is possible. Maybe what you meant is that it eliminates the need for full-precision multiplications by replacing them with multiplications with binary/ternary matrices? 
- The notation is quite confusing. For starters, in Section 2 you mention "a fixed scaling factor A" and I would encourage you to indicate scalars by lower-case letters, vectors by boldface lower-case letters and matrices by boldface upper-case letters. Moreover, it is unclear when calculations are approximate. For instance, in Eq. 1 I believe you need to replace "=" with "\approx". Likewise for the equation in the next to last line on page 2. Lastly, while Eq. 2 seems to be a common way to write down LSTM equations, it is abusive notation.
* Minor Comments
- Abstract: What is ASIC? It is not referenced in Section 6.
- Introduction: What is the justification for calling RNNs over-parameterized? This seems to depend on the task. 
- Introduction; contributions: Here, I would like to see a distinction between gains during training vs test time.
- Section 3.2 comes out of nowhere. You might want to already mention why are introducing batch normalization at this point.
- The boldfacing in Table 1, 2 and 3 is misleading. I understand this is done to highlight the proposed method, but I think commonly boldfacing is used to highlight the best results.
- Figure 2b. What is your hypothesis why BPC actually goes down the longer the sequence is?
- Algorithm 1, line 14: Using the cross-entropy is a specific choice dependent on the task. My understanding is your approach can work with any differentiable downstream loss?