64 during this experiment, effectively doubling the minibatch as we double the number of GPUs.
Although we have the ability to scale to large minibatches, we typically use either 8 or 16 GPUs
during training with a minibatch of 512 or 1024, in order to converge to the best result.
Since all-reduce is critical to the scalability of our training, we wrote our own implementation of
the ring algorithm [48, 65] for higher performance and better stability. Our implementation avoids
extraneous copies between CPU and GPU, and is fundamental to our scalability. We conﬁgure
OpenMPI with the smcuda transport that can send and receive buffers residing in the memory of
two different GPUs by using GPUDirect. When two GPUs are in the same PCI root complex,
this avoids any unnecessary copies to CPU memory. This also takes advantage of tree-structured
interconnects by running multiple segments of the ring concurrently between neighboring devices.
We built our implementation using MPI send and receive, along with CUDA kernels for the elementwise operations.
Table 7 compares the performance of our all-reduce implementation with that provided by OpenMPI
version 1.8.5. We report the time spent in all-reduce for a full training run that ran for one epoch
on our English dataset using a 5 layer, 3 recurrent layer architecture with 2560 hidden units for all
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech--two vastly different languages.
Because it replaces entire pipelines of hand-engineered components with neural
networks, end-to-end learning allows us to handle a diverse variety of speech
including noisy environments, accents and different languages. Key to our
approach is our application of HPC techniques, resulting in a 7x speedup over
our previous system. Because of this efficiency, experiments that previously
took weeks now run in days. This enables us to iterate more quickly to identify
superior architectures and algorithms. As a result, in several cases, our
system is competitive with the transcription of human workers when benchmarked
on standard datasets. Finally, using a technique called Batch Dispatch with
GPUs in the data center, we show that our system can be inexpensively deployed
in an online setting, delivering low latency when serving users at scale.
Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Jingdong Chen Mike Chrzanowski Adam Coates Greg Diamos Erich Elsen Jesse Engel Linxi Fan Christopher Fougner Tony Han Awni Hannun Billy Jun Patrick LeGresley Libby Lin Sharan Narang Andrew Ng Sherjil Ozair Ryan Prenger Jonathan Raiman Sanjeev Satheesh David Seetapun Shubho Sengupta Yi Wang Zhiqian Wang Chong Wang Bo Xiao Dani Yogatama Jun Zhan Zhenyao Zhu