Initial experiments on NORB show that unlike with MNIST where we use deformations, the
CNN needs only 3 to 6 epochs to reach zero validation error. This allows us to quickly run numerous
repetitive experiments with huge networks with hundreds of maps per layer. We decided to use a
CNN with ve hidden layers: layer1, a convolutional layer with 300 maps, kernel size 6 6 and
skipping factors 1 1; layer2, a max-pooling layer over a 2 2 region; layer3, a convolutional layer
with 500 maps, kernel size 4 4, skipping factors 0 0; layer4, a max-pooling layer over a 4 4
region; layer5, a fully connected layer with 500 neurons. The learning rate is initialized by 0.001
and multiplied by 0.95 after every epoch.
Table 2 summarizes the results of four dierent experiments by switching on/o translation as
well as the xed image processing layer. We report the average error rate as well as the standard
deviation of N independent runs with identical architectures but dierent weight initializations.
For the rst experiment without translation and no image processing (IP), an average test error
rate of 7.86% is obtained. With additional translations of at most 5%, the average error rate drops
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber