Technical Report No. IDSIA-01-11 8
Table 2: Average error rates and standard deviations of N runs for a ve hidden layer CNN on
the NORB test set (see text for details).
trans. [%] IP TfbV [%] runs time/epoch [s]
0 no 7.86 0.55 50 1141
5 no 4.71 0.57 50 1563
0 yes 3.94 0.48 50 1658
5 yes 2.530.40 100 2080
similar to the lters automatically created by unsupervised Predictability Minimization (Schmidhuber, 1992) applied to natural images (Schmidhuber et al. , 1996). The rst lter extracts positive
contrast in brightness, whereas the latter extracts negative contrast. Each image from the original
NORB is ltered, consequently the input of the CNN has six maps: the original image plus the
positive and negative contrast for each of the two stereo channels. Using such a pre-processing
layer results in lower average error rates, 3.94% without translation and 2.53% with translation.
This result improves the previous state of the art on NORB (Uetz and Behnke, 2009).
Experience with other image datasets tells us that NORB is unusual. The training set has only
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber