Attention-Based Models for Speech Recognition
Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks including machine translation, handwriting synthesis [1, 2] and image caption generation [3]. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in [2] reaches a competitive 18.7% phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10-times longer (repeated) utterances. Finally, we propose a change to the attention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.
The paper applies attention-based models to speech recognition. The model is extended by considering location awareness, sharpening and smoothing of the scores, and windowing. These extensions overcome the long sentence issues in attention-based models, and achieved the improvement from the straightforward applications of the attention-based models.Quality: This paper describes an original attention-based model, and mathematically explains the extension of a location-awareness score function in Eqs. (5) - (9). The other extensions (sharpening and smoothing) are also mathematically described by considering the long utterance issues. Therefore, the paper has sufficient quality in terms of mathematical/theoretical formulation.Clarity: The paper clearly describes an attention-based model, issues of long sentences, and analysis of experimental discussions. Originality: Although the novelty of this paper is incremental, each step of the incremental extensions is reasonably supported by the experimental results and analysis.Significance: Since the task is relatively small (TIMIT phone recognition), the experimental result is not so significant. Although the proposed method uses a window technique, introducing \alpha requires long-range computations in training and decoding, and the proposed method does not seem to be scalable to large-scale (practical) speech recognition tasks.I summarize the overall Pros. and Cons. as follows: Pros: Attention-based novel architecture provides end-to-end speech recognition. Cons: Scalability.Minor comment: 1. P.2, Section 2: L and T must be explicitly defined.