function then has to be copied back to the GPU for backpropagation. For languages like Mandarin
with large character sets, these matrices have hundreds of millions of entries, making this copy
expensive. Furthermore, we need as much interconnect bandwidth as possible for synchronizing the
gradient updates with data parallelism, so this copy incurs a substantial opportunity cost.
Secondly, although entries in each column of the matrix can be computed in parallel, the number
of entries to calculate in each column depends both on the column and the number of repeated
characters in the utterance label. Due to this complexity, the CPU implementation does not use
SIMD parallelism optimally, making the computation inefﬁcient.
We wrote a GPU-based implementation of CTC in order to overcome these two problems. The key
insight behind our implementation is that we can compute all elements in each column of the 
matrix, rather than just the valid entries. If we do so, Figure 9 shows that invalid elements either
contain a ﬁnite garbage value ( G), or1 (I), when we use a special summation function that adds
probabilities in logspace that discards inputs that are 1. This summation is shown in Figure 9
where arrows incident on a circle are inputs and the result is stored in the circle. However, when we
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech--two vastly different languages.
Because it replaces entire pipelines of hand-engineered components with neural
networks, end-to-end learning allows us to handle a diverse variety of speech
including noisy environments, accents and different languages. Key to our
approach is our application of HPC techniques, resulting in a 7x speedup over
our previous system. Because of this efficiency, experiments that previously
took weeks now run in days. This enables us to iterate more quickly to identify
superior architectures and algorithms. As a result, in several cases, our
system is competitive with the transcription of human workers when benchmarked
on standard datasets. Finally, using a technique called Batch Dispatch with
GPUs in the data center, we show that our system can be inexpensively deployed
in an online setting, delivering low latency when serving users at scale.
Dario Amodei Rishita Anubhai Eric Battenberg Carl Case Jared Casper Bryan Catanzaro Jingdong Chen Mike Chrzanowski Adam Coates Greg Diamos Erich Elsen Jesse Engel Linxi Fan Christopher Fougner Tony Han Awni Hannun Billy Jun Patrick LeGresley Libby Lin Sharan Narang Andrew Ng Sherjil Ozair Ryan Prenger Jonathan Raiman Sanjeev Satheesh David Seetapun Shubho Sengupta Yi Wang Zhiqian Wang Chong Wang Bo Xiao Dani Yogatama Jun Zhan Zhenyao Zhu