in advance.
We solve the power normalization problem by using some statistics from our training set to perform
an adaptive normalization of speech inputs during online transcription. We can solve the other
problems by modifying our network and decoding procedure to produce a model that performs
almost as well while having much lower latency. We focus on our Mandarin system since some
aspects of that system are more challenging to deploy (e.g. the large character set), but the same
techniques could also be applied in English.
In this section, latency refers to the computational latency of our speech system as measured from
the end of an utterance until the transcription is produced. This latency does not include data transmission over the internet, and does not measure latency from the beginning of an utterance until the
ﬁrst transcription is produced. We focus on latency from end of utterance to transcription because it
is important to applications using speech recognition.
7.1 Batch Dispatch
In order to deploy our relatively large deep neural networks at low latency, we have paid special attention to efﬁciency during deployment. Most internet applications process requests individually as
they arrive in the data center. This makes for a straightforward implementation where each request
can be managed by one thread. However, processing requests individually is inefﬁcient computationally, for two main reasons. Firstly, when processing requests individually, the processor must
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech--two vastly different languages.
Because it replaces entire pipelines of hand-engineered components with neural
networks, end-to-end learning allows us to handle a diverse variety of speech
including noisy environments, accents and different languages. Key to our
approach is our application of HPC techniques, resulting in a 7x speedup over
our previous system. Because of this efficiency, experiments that previously
took weeks now run in days. This enables us to iterate more quickly to identify
superior architectures and algorithms. As a result, in several cases, our
system is competitive with the transcription of human workers when benchmarked
on standard datasets. Finally, using a technique called Batch Dispatch with
GPUs in the data center, we show that our system can be inexpensively deployed
in an online setting, delivering low latency when serving users at scale.
Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Jingdong Chen Mike Chrzanowski Adam Coates Greg Diamos Erich Elsen Jesse Engel Linxi Fan Christopher Fougner Tony Han Awni Hannun Billy Jun Patrick LeGresley Libby Lin Sharan Narang Andrew Ng Sherjil Ozair Ryan Prenger Jonathan Raiman Sanjeev Satheesh David Seetapun Shubho Sengupta Yi Wang Zhiqian Wang Chong Wang Bo Xiao Dani Yogatama Jun Zhan Zhenyao Zhu