Exploring Pre Training with Alignments for RNN Transducer Based End to End Speech Recognition
Recently, the recurrent neural network transducer (RNN T) architecture has become an emerging trend in end to end automatic speech recognition research due to its advantages of being capable for online streaming speech recognition. However, RNN T training is made difficult by the huge memory requirements, and complicated neural structure. A common solution to ease the RNN T training is to employ connectionist temporal classification (CTC) model along with RNN language model (RNNLM) to initialize the RNN T parameters. In this work, we conversely leverage external alignments to seed the RNN T model. Two different pre training solutions are explored, referred to as encoder pre training, and whole network pre training respectively. Evaluated on Microsoft 65,000 hours anonymized production data with personally identifiable information removed, our proposed methods can obtain significant improvement. In particular, the encoder pre training solution achieved a 10% and a 8% relative word error rate reduction when compared with random initialization and the widely used CTC+RNNLM initialization strategy, respectively. Our solutions also significantly reduce the RNN T model latency from the baseline.
ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)