Weights (AW) CUDA kernels. The second column stores the number of connections, followed by
corresponding indices of maps in Ln1connected to the current map.
For BP and FP, analogous information about connections is needed. We therefore store backward connections in CBP. AW requires a list of all map connections (see Subsection 3.4), stored
as an array of map index pairs. Dealing with biases in BP kernel requires to know where the
weights of particular connections start; this information is stored in a 2D array WIDX BPof size
MnMn1.
3.2 Forward propagation
A straightforward way of parallelizing FP is to assign a thread block to each map that has to
be computed. For maps bigger than 1024 neurons, the job is further split into smaller blocks by
assigning a block to each line of the map, because the number of threads per block is limited (1024
for GTX 480). A one to one correspondence between threads and the map's neurons is assumed.
Because of weight sharing, threads inside a block can access data in parallel, in particular the
same weights and inputs from the previous layer. Each thread starts by initializing its sum with
the bias, then loops over all map connections, convolving the appropriate patch of the input map
with the corresponding kernel. The output is obtained by passing the sum through a scaled tanh
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber