the bias, then loops over all map connections, convolving the appropriate patch of the input map
with the corresponding kernel. The output is obtained by passing the sum through a scaled tanh
activation function, and then written to device memory.
3.3 Backward propagation
BP of deltas can be done in two ways: by pushing or by pulling. Pushing deltas means taking each
delta from the current layer and computing the corresponding deltas for the previous layer. For
an architecture with shared weights this has the disadvantage of being hard to code. Each delta
from the current layer contributes to many deltas in the previous layer, which translates into a lot
of programming. There are two ways of avoiding this: either writing partial deltas to a separated
block of memory and then putting everything together by calling another kernel (slow because
of a tremendous increase in the number of memory accesses, and the need of another kernel), or
using atomic writes (to avoid data hazards) to update deltas (very slow because many writings
are serialized). We implement pulling deltas, which has almost none of the above speed-limiting
drawbacks, but is a bit more complicated.
Technical Report No. IDSIA-01-11 5
The (uni- or bi-dimensional) thread grid assigns a (bi- or uni-dimensional) thread block to each
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber