Online Hybrid CTC/Attention End to End Automatic Speech Recognition Architecture
Recently, there has been increasing progress in end to end automatic speech recognition (ASR) architecture, which transcribes speech to text without any pre trained alignments. One popular end to end approach is the hybrid Connectionist Temporal Classification (CTC) and attention (CTC/attention) based ASR architecture, which utilizes the advantages of both CTC and attention. The hybrid CTC/attention ASR systems exhibit performance comparable to that of the conventional deep neural network (DNN) hidden Markov model (HMM) ASR systems. However, how to deploy hybrid CTC/attention systems for online speech recognition is still a non trivial problem. This article describes our proposed online hybrid CTC/attention end to end ASR architecture, which replaces all the offline components of conventional CTC/attention ASR architecture with their corresponding streaming components. Firstly, we propose stable monotonic chunk wise attention (sMoChA) to stream the conventional global attention, and further propose monotonic truncated attention (MTA) to simplify sMoChA and solve the training and decoding mismatch problem of sMoChA. Secondly, we propose truncated CTC (T CTC) prefix score to stream CTC prefix score calculation. Thirdly, we design dynamic waiting joint decoding (DWJD) algorithm to dynamically collect the predictions of CTC and attention in an online manner. Finally, we use latency controlled bidirectional long short term memory (LC BLSTM) to stream the widely used offline bidirectional encoder network. Experiments with LibriSpeech English and HKUST Mandarin tasks demonstrate that, compared with the offline CTC/attention model, our proposed online CTC/attention model improves the real time factor in human computer interaction services and maintains its performance with moderate degradation. To the best of our knowledge, this is the first work to provide the full stack online solution for CTC/attention end to end ASR architecture.
IEEE/ACM Transactions on Audio, Speech, and Language Processing