shown to improve speech systems [21, 26]. Techniques used for data augmentation in speech range
from simple noise addition [26] to complex perturbations such as simulating changes to the vocal
tract length and rate of speech of the speaker [31, 35].
Existing speech systems can also be used to bootstrap new data collection. In one approach, the
authors use one speech engine to align and ﬁlter a thousand hours of read speech [46]. In another
approach, a heavy-weight ofﬂine speech recognizer is used to generate transcriptions for tens of
thousands of hours of speech [33]. This is then passed through a ﬁlter and used to re-train the recognizer, resulting in signiﬁcant performance gains. We draw inspiration from these past approaches in
3
bootstrapping larger datasets and data augmentation to increase the effective amount of labeled data
for our system.
3 Model Architecture
A simple multi-layer model with a single recurrent layer cannot exploit thousands of hours of labelled speech. In order to learn from datasets this large, we increase the model capacity via depth.
We explore architectures with up to 11 layers including many bidirectional recurrent layers and convolutional layers. These models have nearly 8 times the amount of computation per data example as
the models in Deep Speech 1 making fast optimization and computation critical. In order to optimize
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech--two vastly different languages.
Because it replaces entire pipelines of hand-engineered components with neural
networks, end-to-end learning allows us to handle a diverse variety of speech
including noisy environments, accents and different languages. Key to our
approach is our application of HPC techniques, resulting in a 7x speedup over
our previous system. Because of this efficiency, experiments that previously
took weeks now run in days. This enables us to iterate more quickly to identify
superior architectures and algorithms. As a result, in several cases, our
system is competitive with the transcription of human workers when benchmarked
on standard datasets. Finally, using a technique called Batch Dispatch with
GPUs in the data center, we show that our system can be inexpensively deployed
in an online setting, delivering low latency when serving users at scale.
Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Jingdong Chen Mike Chrzanowski Adam Coates Greg Diamos Erich Elsen Jesse Engel Linxi Fan Christopher Fougner Tony Han Awni Hannun Billy Jun Patrick LeGresley Libby Lin Sharan Narang Andrew Ng Sherjil Ozair Ryan Prenger Jonathan Raiman Sanjeev Satheesh David Seetapun Shubho Sengupta Yi Wang Zhiqian Wang Chong Wang Bo Xiao Dani Yogatama Jun Zhan Zhenyao Zhu