not measurably change recognition accuracy. Because deployment does not require any updates to
the network weights, it is far less sensitive to numerical precision than training. Using half-precision
arithmetic saves memory space and bandwidth, which is especially useful for deployment, since
RNN evaluation is dominated by the cost of caching and streaming the weight matrices.
As seen in Section 7.1, the batch size during deployment is much smaller than in training. We
found that standard BLAS libraries are inefﬁcient at this batch size. To overcome this, we wrote our
own half-precision matrix-matrix multiply kernel. For 10 simultaneous streams over 90 percent of
batches are for N4, a regime where the matrix multiply will be bandwidth bound. We store the A
matrix transposed to maximize bandwidth by using the widest possible vector loads while avoiding
transposition after loading. Each warp computes four rows of output for all Noutput columns. Note
that forN4theBmatrix ﬁts entirely in the L1 cache. This scheme achieves 90 percent of peak
bandwidth for N4but starts to lose efﬁciency for larger Nas theBmatrix stops ﬁtting into the
L1 cache. Nonetheless, it continues to provide improved performance over existing libraries up to
N= 10 .
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech--two vastly different languages.
Because it replaces entire pipelines of hand-engineered components with neural
networks, end-to-end learning allows us to handle a diverse variety of speech
including noisy environments, accents and different languages. Key to our
approach is our application of HPC techniques, resulting in a 7x speedup over
our previous system. Because of this efficiency, experiments that previously
took weeks now run in days. This enables us to iterate more quickly to identify
superior architectures and algorithms. As a result, in several cases, our
system is competitive with the transcription of human workers when benchmarked
on standard datasets. Finally, using a technique called Batch Dispatch with
GPUs in the data center, we show that our system can be inexpensively deployed
in an online setting, delivering low latency when serving users at scale.
Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Jingdong Chen Mike Chrzanowski Adam Coates Greg Diamos Erich Elsen Jesse Engel Linxi Fan Christopher Fougner Tony Han Awni Hannun Billy Jun Patrick LeGresley Libby Lin Sharan Narang Andrew Ng Sherjil Ozair Ryan Prenger Jonathan Raiman Sanjeev Satheesh David Seetapun Shubho Sengupta Yi Wang Zhiqian Wang Chong Wang Bo Xiao Dani Yogatama Jun Zhan Zhenyao Zhu