from Table 1 reaches 2.42%, 0.97% and 0.48% after one, three and seventeen epochs, respectively.
On the other hand, the network with 4 instead of 7 hidden layers reaches 4.71%, 1.58%, 0.68%
after one, three and seventeen epochs, achieving a test error below 0.50% after only 34 epochs.
This shows once more that deep networks, contrary to common belief, can be trained successfully
by back-propagation. Despite the numerous free parameters, deep networks seem to learn faster
(better recognition rates after fewer epochs) than shallow ones.
We consider MNIST an almost solved problem. The remaining errors stem from digits that
are ambiguous or miss parts.
4.2 Experiments on NORB
NORB contains stereo images of 3D objects. Hence there are two maps on the input layer.
Rotation, scaling, shearing and elastic distortions seem to have a negative impact on generalization.
These deformations improve recognition rates for digits that are intrinsically 2D (Cire san et al. ,
2010), but seem inadequate for 3D objects.
Initial experiments on NORB show that unlike with MNIST where we use deformations, the
CNN needs only 3 to 6 epochs to reach zero validation error. This allows us to quickly run numerous
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber