test error of 0.27%. The best previous CNN result on MNIST is 0.40% (Simard et al. , 2003).
A 0.35% error rate was recently also obtained by a big, deep MLP (Cire san et al. , 2010) with
many more free parameters. Deeper nets require more computation time to complete an epoch,
but we observe that they also need fewer epochs to achieve good test errors. The deepest CNN
Technical Report No. IDSIA-01-11 7
Table 1: Error rates on MNIST test set for randomly connected CNNs with 2 to 6 convolutional
layers with M Maps and an optional fully connected layer with N neurons. Various kernel sizes
and skipping factors were used.
#M, #N bT TfbV
in Hidden Layers [%] [%]
20M-60M 0.95 1.02
20M-60M-150N 0.50 0.55
20M-60M-100M-150N 0.33 0.38
20M-40M-60M-80M-100M-120M-150N 0.27 0.35
from Table 1 reaches 2.42%, 0.97% and 0.48% after one, three and seventeen epochs, respectively.
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber