By how much do they outperform existing state-of-the-art model on end-to-end Speech recognition?s 
10%
Attention-based encoder-decoder model has achieved impressive results for both automatic speech recognition (ASR) and text-to-speech (TTS) tasks. This approach takes advantage of the memorization capacity of neural networks to learn the mapping from the input sequence to the output sequence from scratch, without the assumption of prior knowledge such as the alignments. However, this model is prone to overfitting, especially when the amount of training data is limited. Inspired by SpecAugment and BERT, in this paper, we propose a semantic mask based regularization for training such kind of end-to-end (E2E) model. The idea is to mask the input features corresponding to a particular output token, e.g., a word or a word-piece, in order to encourage the model to fill the token based on the contextual information. While this approach is applicable to the encoder-decoder framework with any type of neural network architecture, we study the transformer-based model for ASR in this work. We perform experiments on Librispeech 960h and TedLium2 data sets, and achieve the state-of-the-art performance on the test set in the scope of E2E models.
End-to-end (E2E) acoustic models, particularly with the attention-based encoder-decoder framework BIBREF0, have achieved a competitive recognition accuracy in a wide range of speech datasets BIBREF1. This model directly learns the mapping from the input acoustic signals to the output transcriptions without decomposing the problems into several different modules such as lexicon modeling, acoustic modeling and language modeling as in the conventional hybrid architecture. While this kind of E2E approach significantly simplifies the speech recognition pipeline, the weakness is that it is difficult to tune the strength of each component. One particular problem from our observations is that the attention based E2E model tends to make grammatical errors, which indicates that the language modeling power of the model is weak, possibly due to the small amount of training data, or the mismatch between the training and evaluation data. However, due to the jointly model approach in the attention model, it is unclear how to improve the strength of the language modeling power, i.e., attributing more weights to the previous output tokens in the decoder, or to improve the strength of the acoustic modeling power, i.e., attributing more weights to the context vector from the encoder. While an external language model may be used to mitigate the weakness of the language modeling power of an attention-based E2E model, by either re-scoring the hypothesis or through shallow or deep fusion BIBREF2, the improvements are usually limited, and it incurs additional computational cost. Inspired by SpecAgument BIBREF3 and BERT BIBREF4, we propose a semantic mask approach to improve the strength of the language modeling power in the attention-based E2E model, which, at the same time, improves the generalization capacity of the model as well. Like SpecAugment, this approach masks out partial of the acoustic features during model training. However, instead of using a random mask as in SpecAugment, our approach masks out the whole patch of the features corresponding to an output token during training, e.g., a word or a word-piece. The motivation is to encourage the model to fill in the missing token (or correct the semantic error) based on the contextual information with less acoustic evidence, and consequently, the model may have a stronger language modeling power and is more robust to acoustic distortions. In principle, our approach is applicable to the attention-based E2E framework with any type of neural network encoder. To constrain our research scope, we focus on the transformer architecture BIBREF5, which is originally proposed for neural machine translation. Recently, it has been shown that the transformer model can achieve competitive or even higher recognition accuracy compared with the recurrent neural network (RNN) based E2E model for speech recognition BIBREF6. Compared with RNNs, the transformer model can capture the long-term correlations with a computational complexity of $O(1)$, instead of using many steps of back-propagation through time (BPTT) as in RNNs. We evaluate our transformer model with semantic masking on Librispeech and TedLium datasets. We show that semantic masking can achieve significant word error rate reduction (WER) on top of SpecAugment, and we report the lowest WERs on the test sets of the Librispeech corpus with an E2E model.