Fully Neural Network Based Speech Recognition on Mobile and Embedded Devices
Real-time automatic speech recognition (ASR) on mobile and embedded devices has been of great interests for many years. We present real-time speech recognition on smartphones or embedded systems by employing recurrent neural network (RNN) based acoustic models, RNN based language models, and beam-search decoding. The acoustic model is end-to-end trained with connectionist temporal classification (CTC) loss. The RNN implementation on embedded devices can suffer from excessive DRAM accesses because the parameter size of a neural network usually exceeds that of the cache memory and the parameters are used only once for each time step. To remedy this problem, we employ a multi-time step parallelization approach that computes multiple output samples at a time with the parameters fetched from the DRAM. Since the number of DRAM accesses can be reduced in proportion to the number of parallelization steps, we can achieve a high processing speed. However, conventional RNNs, such as long short-term memory (LSTM) or gated recurrent unit (GRU), do not permit multi-time step parallelization. We construct an acoustic model by combining simple recurrent units (SRUs) and depth-wise 1-dimensional convolution layers for multi-time step parallelization. Both the character and word piece models are developed for acoustic modeling, and the corresponding RNN based language models are used for beam search decoding. We achieve a competitive WER for WSJ corpus using the entire model size of around 15MB and achieve real-time speed using only a single core ARM without GPU or special hardware.
Summary: This paper proposes an acoustic model by combining simple recurrent units (SRUs) and simple 1-D convolution layers for multi-time step parallelization, which significantly improves the speed by overcoming the step-by-step constraint of RNNs. The proposed AM performs substantially better than SRUs alone and even better than LSTMs with the same number of parameters. It is combined with an RNN LM as a fully neural network-based speech recognizer. Word pieces are investigated as the model output targets in addition to graphemes, for further speedup in decoding.  Quality: Strengths: - The idea of SRU + 1-D conv is simple but the improvement is impressive. - The multi-time step parallel processing for AM is shown to achieve nice speedup on a single core (6x with 8 parallel steps). - RNN LM with multi-stream processing is shown to be efficient enough to be used in beam search decoding, where it is typically used as second-pass rescoring due to efficiency constraint. - The paper shows a final model with total size 30MB is able to run in real time with a single core ARM.  Weaknesses: - Do all the baseline LSTM models in the paper use recurrent projection units (Sak et al. 2014, Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling)? It should be easy to implement and it is effective to improve performance with the same number of parameters. - LSTM + 1-D conv should be used as the baseline in Table 3 and 5 since it is shown to be better than LSTM alone in Table 1. - It is kind of surprising that SRU + 1-D conv achieves better accuracy than LSTM + 1-D conv. It is worth discussing the reason / hypothesis in the paper. - In Table 6, it would be useful to also measure the computation time of LSTM AM for comparison. - The proposed system should be compared to a standard LSTM AM + WFST decoder graph with a (pruned) n-gram LM, which is shown to be efficient for embedded speech recognition in previous works (e.g. McGraw et al. 2016, Personalized speech recognition on mobile devices), especially now that the RNN LM is the computation bottleneck of the entire system as is shown in Figure 2. In fact, the speed of RNN LM could be further improved by 8-bit quantization, which typically does not degrade the WER.  Clarity: In general, the presentation in the paper is clear and well-organized, except for the following questions: - In Table 2, what is the system performance with HCLM? - Does the system in Table 6 use characters or word pieces? Both should be shown for comparison. - How much is the WER different between different beam width in Figure 2? Does using half-precision in LM hurt WER? - It would be clearer to mention in Table 2 that 3.60 (for Deep Speech 2) is WER (instead of CER).  Originality: The problem is well-motivated: multi-time step parallel processing for AM would be very useful for embedded system recognition, but SRU alone does not work. It is novel to find that simply combining 1-D conv with SRU substantially improves the quality of SRU, leading to better performance than LSTM. Other tricks and analyses in the paper to improve speed and accuracy are also useful.   Significance: The results are very encouraging in terms of both accuracy and speed. Both the idea and the experiments would be of interest to the field and motivate future work.  It is a good submission. I recommend an accept, although the issues mentioned above should be addressed.