Mutch and Lowe, 2008), whose lters are xed, in contrast to those of Convolutional Neural
Networks (CNNs) (LeCun et al. , 1998; Behnke, 2003; Simard et al. , 2003), whose weights (lters)
are randomly initialized and changed in a supervised way using back-propagation (BP).
Despite the hardware progress of the past decades, computational speed is still a limiting
factor for CNN architectures characterized by many building blocks typically set by trial and
error. To systematically test the impact of various architectures on classication performance,
we present a fast CNN implementation on Graphics Processing Units (GPUs). Previous GPU
implementations of CNNs (Chellapilla et al. , 2006; Uetz and Behnke, 2009) were hard-coded to
satisfy GPU hardware constraints, whereas our implementation is 
exible and fully online (i.e.,
Technical Report No. IDSIA-01-11 2
weight updates after each image). It allows for training large CNNs within days instead of months,
such that we can investigate the in
uence of various structural parameters by exploring large
parameter spaces (Pinto et al. , 2009) and performing error analysis on repeated experiments.
We evaluate various networks on the handwritten digit benchmark MNIST (LeCun et al. , 1998)
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber