into a 1D feature vector. The top layer is always fully connected, with one output unit per class
label.
Figure 1: Architecture of a convolutional neural network. In this case, the convolutional layers
are fully connected. Both convolutional layers use a kernel of 5 x 5 and skipping factors of 1.
3 GPU implementation
The latest generation of NVIDIA GPUs, the 400 and 500 series (we use GTX 480 & GTX 580), has
many advantages over older GPUs, most notably the presence of a R/W L2 global cache for device
memory. This permits faster programs and simplies writing the code. In fact, the corresponding
transfer of complexity into hardware alleviates many software and optimization problems. Our
experiments show that the CNN program becomes 2-3 times faster just by switching from GTX
285 to GTX 480.
Technical Report No. IDSIA-01-11 4
Manual optimization of CUDA code is very time-consuming and error prone. We optimize for
the new architecture, relying on the L2 cache for many of the device memory accesses, instead of
manually writing code that uses textures and shared memory. Code obtained by this pragmatic
strategy is fast enough. We use the following types of optimization: pre-computed expressions,
unrolled loops within template kernels, strided matrices to obtain coalesced memory accesses and
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber