3.4 Adjusting weights
FP and BP have a grid on the list of maps, but the AW thread grid is on the list of kernels (lters)
between maps of two consecutive layers. The 1D grid has a block for each connection between two
maps. Thread blocks are 2D, with a corresponding thread for every kernel weight. The bias weight
is included as an entire row of threads, thus requiring thread blocks to have ( Kx+1)Kythreads.
Most of the time these additional Kythreads will do nothing, thread (0,0) being activated only
for blocks that have to process the bias.
4 Experiments
We use a system with a Core i7-920 (2.66GHz), 12 GB DDR3 and four graphics cards: 2 x
GTX 480 and 2 x GTX 580. The correctness of the CPU version is checked by comparing the
analytical gradient with its nite dierence approximation. On GPU this is not possible because
all computations are performed with single precision 
oating point numbers. Hence the GPU
implementation's correctness is checked by comparing its results to those of a randomly initialized
net after training it for several epochs on the more accurate CPU version. Obtaining identical
results after trillions of operations is a strong indication of correctness.
Technical Report No. IDSIA-01-11 6
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber