(Test for best Validation - TfbV). The best test error (bT) is also listed for all experiments. The
reported computation times per epoch include training, validation and testing as well as all data
transfers.
4.1 Experiments on MNIST
For the MNIST dataset the networks are trained on deformed images, continually generated in
on-line fashion. Ane (translation, rotation, scaling, horizontal shearing) and elastic deformations
(Simard et al. , 2003) are combined. We use a variable learning rate that shrinks by a multiplicative
constant after each epoch, from 103down to 3 105after 500 epochs.
Fully connected convolutional layers lead to an exploding number of network connections and
weights, making training of big and deep CNNs for hundreds of epochs impractical even on GPUs.
Partial connectivity alleviates this problem and is also biologically more plausible. We reduce
the number of connections between convolutional layers in a random way. Table 1 lists results
of various networks with 2 to 7 hidden layers with random connections. Additional layers result
in better networks, the best one achieving a test error of 0.35% for best validation and a best
test error of 0.27%. The best previous CNN result on MNIST is 0.40% (Simard et al. , 2003).
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber