We evaluate various networks on the handwritten digit benchmark MNIST (LeCun et al. , 1998)
and two image classication benchmarks: NORB (LeCun et al. , 2004) and CIFAR10 (Krizhevsky,
2009).
2 Convolutional neural networks
CNNs are hierarchical neural networks whose convolutional layers alternate with subsampling
layers, reminiscent of simple and complex cells in the primary visual cortex (Wiesel and Hubel,
1959). CNNs vary in how convolutional and subsampling layers are realized and how the nets
are trained. The CNN architecture considered in this study diers from the one of Simard et al.
(2003) in the sense that after each CNN-layer an optional max-pooling layer (Scherer et al. , 2010)
can be used. Here we give a complete description of this independent implementation (Fig. 1).
2.1 Image processing layer
The image processing layer is an optional pre-processing layer of predened lters that are kept
xed during training. Thus additional information besides the raw input image can be provided
to the network, such as edges and gradients. In particular, we nd that a contrast-extracting layer
(Fukushima, 2003) helps to improve the recognition rate for NORB.
2.2 Convolutional layer
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber