4.4 Speedup factor of GPU code
The GPU code scales well with network size. For small nets the speedup is small (but still over 10)
since they t better inside the CPU cache, and GPU resources are underutilized. For huge nets
(ex: Table 2) the GPU implementation is more than 60 times faster than a compiler-optimized
CPU version. Given the 
exibility of our GPU version, this is a signicant speedup. One epoch
takes 35 GPU minutes but more than 35 CPU hours.
5 Conclusion
We presented high-performance GPU-based CNN variants trained by on-line gradient descent, with
sparse random connectivity, computationally more ecient and biologically more plausible than
fully connected CNNs. Principal advantages include state-of-the-art generalization capabilities,
great 
exibility and speed. All structural CNN parameters such as input image size, number of
hidden layers, number of maps per layer, kernel sizes, skipping factors and connection tables are
adaptable to any particular application. We applied our networks to benchmark datasets for digit
recognition (MNIST), 3D object recognition (NORB), and natural images (CIFAR10). On MNIST
the best network achieved a recognition test error rate of 0.35%, on NORB 2.53% and on CIFAR10
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.
Dan C. Cireşan Ueli Meier Jonathan Masci Luca M. Gambardella Jürgen Schmidhuber