results after trillions of operations is a strong indication of correctness.
Technical Report No. IDSIA-01-11 6
Figure 2: Back propagating deltas. A connection between two maps from two consecutive layers
is displayed. The map in Ln1has 29 x 29 neurons; the map in Lnhas 13 x 13 neurons. They
are linked through a 5 x 5 kernel K. Skipping factors of Sx= 1 andSy= 1 are assumed. Arrows
and colors depict the correspondence between neurons in Ln1and their sources in Ln.
The implemented CNN's plain feed-forward architecture is trained using on-line gradient descent. All images from the training set are used for training and also for validation. If deformations
are enabled, only the images from the training set will be deformed. Weights are initialized according to a uniform random distribution in the range [ 0:05;0:05]. Each neuron's activation
function is a scaled hyperbolic tangent: y(a) = 1:7159 tanh(0 :6666a) (LeCun et al. , 1998).
We pick the trained CNN with the lowest validation error, and evaluate it on the test set
(Test for best Validation - TfbV). The best test error (bT) is also listed for all experiments. The
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber