  We propose a multitask training method for attention-based end-to-end speech
recognition models. We regularize the decoder in a listen, attend, and spell
model by multitask training it on both audio-text and text-only data. Trained
on the 100-hour subset of LibriSpeech, the proposed method, without requiring
an additional language model, leads to an 11% relative performance improvement
over the baseline and approaches the performance of language model shallow
fusion on the test-clean evaluation set. We observe a similar trend on the
whole 960-hour LibriSpeech training set. Analyses of different types of errors
and sample output sentences demonstrate that the proposed method can
incorporate language level information, suggesting its effectiveness in
real-world applications.

Computation and Language