rate of 7.86% is obtained. With additional translations of at most 5%, the average error rate drops
to 4.71%, contradicting the common belief that CNNs are translation invariant. These results are
on par or better than others in the literature: 5.90% error rate for a combination of CNNs and
SVMs (LeCun et al. , 2004) and 5.20% error rate for restricted Boltzman machines (Nair and
Hinton, 2009).
The best previously published result on NORB (2.87%) was obtained by a hierarchical neural
network which to every convolutional layer provides a subsampled version plus edge information
of the original image (Uetz and Behnke, 2009). This motivated us to implement a pre-processing
layer with xed lters. We tried simple edge masks (Sobel, Scharr) but obtained best results with a
contrast-extraction layer (Fukushima, 2003) realized by Mexican hat-shaped lters of size 21 21,
one with a concentric on-center receptive eld and one with a concentric o-center receptive eld,
Technical Report No. IDSIA-01-11 8
Table 2: Average error rates and standard deviations of N runs for a ve hidden layer CNN on
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber