This system has two main components—a deep learning library written in C ++, along with a highperformance linear algebra library written in both CUDA and C ++. Our optimized software, running
on dense compute nodes with 8 Titan X GPUs per node, allows us to sustain 24 single-precision
teraFLOP/second when training a single model on one node. This is 45% of the theoretical peak
computational throughput of each node. We also can scale to multiple nodes, as outlined in the next
subsection.
4.1 Scalability and Data-Parallelism
We use the standard technique of data-parallelism to train on multiple GPUs using synchronous
SGD. Our most common conﬁguration uses a minibatch of 512on8GPUs. Our training pipeline
12
binds one process to each GPU. These processes then exchange gradient matrices during the backpropagation by using all-reduce, which exchanges a matrix between multiple processes and sums
the result so that at the end, each process has a copy of the sum of all matrices from all processes.
We ﬁnd synchronous SGD useful because it is reproducible and deterministic. We have found
that the appearance of non-determinism in our system often signals a serious bug, and so having
reproducibility as a goal has greatly facilitates debugging. In contrast, asynchronous methods such
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech--two vastly different languages.
Because it replaces entire pipelines of hand-engineered components with neural
networks, end-to-end learning allows us to handle a diverse variety of speech
including noisy environments, accents and different languages. Key to our
approach is our application of HPC techniques, resulting in a 7x speedup over
our previous system. Because of this efficiency, experiments that previously
took weeks now run in days. This enables us to iterate more quickly to identify
superior architectures and algorithms. As a result, in several cases, our
system is competitive with the transcription of human workers when benchmarked
on standard datasets. Finally, using a technique called Batch Dispatch with
GPUs in the data center, we show that our system can be inexpensively deployed
in an online setting, delivering low latency when serving users at scale.
Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Jingdong Chen Mike Chrzanowski Adam Coates Greg Diamos Erich Elsen Jesse Engel Linxi Fan Christopher Fougner Tony Han Awni Hannun Billy Jun Patrick LeGresley Libby Lin Sharan Narang Andrew Ng Sherjil Ozair Ryan Prenger Jonathan Raiman Sanjeev Satheesh David Seetapun Shubho Sengupta Yi Wang Zhiqian Wang Chong Wang Bo Xiao Dani Yogatama Jun Zhan Zhenyao Zhu