Table 6 shows that an external language model helps both English and Mandarin speech systems.
The relative improvement given by the language model drops from 48% to 36% in English and 27%
to 23% in Mandarin, as we go from a model with 5 layers and 1 recurrent layer to a model with 9
layers and 7 recurrent layers. We hypothesize that the network builds a stronger implicit language
model with more recurrent layers.
The relative performance improvement from a language model is higher in English than in Mandarin.
We attribute this to the fact that a Chinese character represents a larger block of information than
an English character. For example, if we output directly to syllables or words in English, the model
would make fewer spelling mistakes and the language model would likely help less.
3.9 Adaptation to Mandarin
The techniques that we have described so far can be used to build an end-to-end Mandarin speech
recognition system that outputs Chinese characters directly. This precludes the need to construct a
pronunciation model, which is often a fairly involved component for porting speech systems to other
languages [59]. Direct output to characters also precludes the need to explicitly model language
speciﬁc pronunciation features. For example we do not need to model Mandarin tones explicitly, as
some speech systems must do [59, 45].
The only architectural changes we make to our networks are due to the characteristics of the Chinese
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech--two vastly different languages.
Because it replaces entire pipelines of hand-engineered components with neural
networks, end-to-end learning allows us to handle a diverse variety of speech
including noisy environments, accents and different languages. Key to our
approach is our application of HPC techniques, resulting in a 7x speedup over
our previous system. Because of this efficiency, experiments that previously
took weeks now run in days. This enables us to iterate more quickly to identify
superior architectures and algorithms. As a result, in several cases, our
system is competitive with the transcription of human workers when benchmarked
on standard datasets. Finally, using a technique called Batch Dispatch with
GPUs in the data center, we show that our system can be inexpensively deployed
in an online setting, delivering low latency when serving users at scale.
Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Jingdong Chen Mike Chrzanowski Adam Coates Greg Diamos Erich Elsen Jesse Engel Linxi Fan Christopher Fougner Tony Han Awni Hannun Billy Jun Patrick LeGresley Libby Lin Sharan Narang Andrew Ng Sherjil Ozair Ryan Prenger Jonathan Raiman Sanjeev Satheesh David Seetapun Shubho Sengupta Yi Wang Zhiqian Wang Chong Wang Bo Xiao Dani Yogatama Jun Zhan Zhenyao Zhu