strategy is fast enough. We use the following types of optimization: pre-computed expressions,
unrolled loops within template kernels, strided matrices to obtain coalesced memory accesses and
registers wherever possible. Additional manual optimizations are possible in case future image
classication problems will require even more computing power.
3.1 Data structures
Both outputs yand deltasof layerLnare 2D strided. Their original size is MxMM y, but they
are horizontally strided with a pitch of 32 
oats (we use this stride for all 2D data), resulting in
coalesced memory accesses. The vertical stride avoids additional bounding tests in CUDA kernels.
All connections between maps of consecutive layers Ln1andLnare stored in matrix Cn.
Each row of Cncontains all connections that feed into a particular map in layer Ln. Because
we aim for a 
exible architecture with partially connected layers, in the rst column we store the
number of previous connections. This index is useful for Forward Propagation (FP) and Adjusting
Weights (AW) CUDA kernels. The second column stores the number of connections, followed by
corresponding indices of maps in Ln1connected to the current map.
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber