We propose a novel way of reducing the number of parameters in the storagehungry fully connected layers of a neural network by using pre-defined sparsity, where the majority of connections are absent prior to starting training. Our results indicate that convolutional neural networks can operate without any loss of accuracy at less than 0.5% classification layer connection density, or less than 5% overall network connection density. We also investigate the effects of pre-defining the sparsity of networks with only fully connected layers. Based on our sparsifying technique, we introduce the 'scatter' metric to characterize the quality of a particular connection pattern. As proof of concept, we show results on CIFAR, MNIST and a new dataset on classifying Morse code symbols, which highlights some interesting trends and limits of sparse connection patterns.
This paper examines sparse connection patterns in upper layers of convolutional image classification networks.  Networks with very few connections in the upper layers are experimentally determined to perform almost as well as those with full connection masks.  Heuristics for distributing connections among windows/groups and a measure called "scatter" are introduced to construct the connectivity masks, and evaluated experimentally on CIFAR-10 and -100, MNIST and Morse code symbols.
While it seems clear in general that many of the connections are not needed and can be made sparse (Figures 1 and 2), I found many parts of this paper fairly confusing, both in how it achieves its objectives, as well as much of the notation and method descriptions.  I've described many of the points I was confused by in more detailed comments below.
Detailed comments and questions:
The distribution of connections in "windows" are first described to correspond to a sort of semi-random spatial downsampling, to get different views distributed over the full image.  But in the upper layers, the spatial extent can be very small compared to the image size, sometimes even 1x1 depending on the network downsampling structure.  So are do the "windows" correspond to spatial windows, and if so, how?  Or are they different (maybe arbitrary) groupings over the feature maps?
Also a bit confusing is the notation "conv2", "conv3", etc.  These names usually indicate the name of a single layer within the network (conv2 for the second convolutional layer or series of layers in the second spatial size after downsampling, for example).  But here it seems just to indicate the number of "CL" layers: 2.  And p.1 says that the "CL" layers are those often referred to as "FC" layers, not "conv" (though they may be convolutionally applied with spatial 1x1 kernels).
The heuristic for spacing connections in windows across the spatial extent of an image makes intuitive sense, but I'm not convinced this will work well in all situations, and may even be sub-optimal for the examined datasets.  For example, to distinguish MNIST 1 vs 7 vs 9, it is most important to see the top-left:  whether it is empty, has a horizontal line, or a loop.  So some regions are more important than others, and the top half may be more important than an equally spaced global view.  So the description of how to space connections between windows makes some intuitive sense, but I'm unclear on whether other more general connections might be even better, including some that might not be as easily analyzed with the "scatter" metric described.
Another broader question I have is in the distinction between lower and upper layers (those referred to as "feature extracting" and "classification" in this paper).  It's not clear to me that there is a crisply defined difference here (though some layers may tend to do more of one or the other function, such as we might interpret).  So it seems that expanding the investigation to include all layers, or at least more layers, would be good:  It might be that more of the "classification" function is pushed down to lower layers, as the upper layers are reduced in size.  How would they respond to similar reductions?
I'm also unsure why on p.6 MNIST uses 2d windows, while CIFAR uses 3d --- The paper mentions the extra dimension is for features, but MNIST would have a features dimension as well at this stage, I think?  I'm also unsure whether the windows are over spatial extent only, or over features.