Deep learning research over the past years has shown that by increasing the scope or difficulty of the learning problem over time, increasingly complex learning problems can be addressed. We study incremental learning in the context of sequence learning, using generative RNNs in the form of multi-layer recurrent Mixture Density Networks. While the potential of incremental or curriculum learning to enhance learning is known, indiscriminate application of the principle does not necessarily lead to improvement, and it is essential therefore to know which forms of incremental or curriculum learning have a positive effect. This research contributes to that aim by comparing three instantiations of incremental or curriculum learning. We introduce Incremental Sequence Learning, a simple incremental approach to sequence learning. Incremental Sequence Learning starts out by using only the first few steps of each sequence as training data. Each time a performance criterion has been reached, the length of the parts of the sequences used for training is increased. To evaluate Incremental Sequence Learning and comparison methods, we introduce and make available a novel sequence learning task and data set: predicting and classifying MNIST pen stroke sequences, where the familiar handwritten digit images have been transformed to pen stroke sequences representing the skeletons of the digits. We find that Incremental Sequence Learning greatly speeds up sequence learning and reaches the best test performance level of regular sequence learning 20 times faster, reduces the test error by 74%, and in general performs more robustly; it displays lower variance and achieves sustained progress after all three comparison methods have stopped improving. The two other instantiations of curriculum learning do not result in any noticeable improvement. A trained sequence prediction model is also used in transfer learning to the task of sequence classification, where it is found that transfer learning realizes improved classification performance compared to methods that learn to classify from scratch.
1 INTRODUCTION
1.1 INCREMENTAL LEARNING, TRANSFER LEARNING, AND REPRESENTATION LEARNING
Deep learning research over the past years has shown that by increasing the scope or difficulty of the learning problem over time, increasingly complex learning problems can be addressed. This principle has been described as Incremental learning by Elman (1991), and has a long history. Schlimmer and Granger (1986) described a pseudo-connectionist distributed concept learning approach involving incremental learning. Elman (1991) defined Incremental Learning as an approach where the training data is not presented all at once, but incrementally; see also Elman (1993). Giraud-Carrier (2000) defines Incremental Learning as follows: “A learning task is incremental if the training examples used to solve it become available over time, usually one at a time.“ Bengio et al. (2009) introduced the framework of Curriculum Learning. The central idea behind this approach is that a learning system is guided by presenting gradually more and/or more complex concepts. A formal definition is provided specifying that the distribution over examples converges monotonically towards the target training
distribution, and that the entropy of the distributions visited over time, and hence the diversity of training examples, increases.
An extension of the notion of incremental learning is to also let the learning task vary over time. This approach, known as Transfer Learning or Inductive Transfer, was first described by Pratt (1993). Thrun (1996) reported improved generalization performance for lifelong learning and described representation learning, whereas Caruana (1997) considered a Multitask learning setup where tasks are learned in parallel while using a shared representation. In coevolutionary algorithms, the coevolution of representations with solutions that employ them, see e.g. Moriarty (1997); de Jong and Oates (2002), provides another approach to representation learning. Representation learning can be seen as a special form of transfer learning, where one goal is to learn adequate representations, and the other goal, addressed in parallel or sequentially, is to use these representations to address the learning problem.
Several of the recent successes of deep learning can be attributed to representation learning and incremental learning. Bengio et al. (2013) provide a review and insightful discussion of representation learning. Parisotto et al. (2015) report experiments with transfer learning across Atari 2600 arcade games where up to 5 million frames of training time in each game are saved. More recently, successful transfer of robot learning from the virtual to the real world was achieved using transfer learning, see Rusu et al. (2016). And at the annual ImageNet Large-Scale Visual Recognition Challenge (ILSVRC), the depth of networks has steadily increased over the years, so far leading up to a network of 152 layers for the winning entry in the ILSVRC 2015 classification task; see He et al. (2015).
1.2 SEQUENCE LEARNING
We study incremental learning in the context of sequence learning. The aim in sequence learning is to predict, given a step of the sequence, what the next step will be. By iteratively feeding the predicted output back into the network as the next input, the network can be used to produce a complete sequences of variable length. For a discussion of variants of sequence learning problems, see Sun and Giles (2001); a more recent treatment covering recurrent neural networks as used here is provided by Lipton (2015).
An interesting challenge in sequence learning is that for most sequence learning problems of interest, the next step in a sequence does not follow unambiguously from the previous step. If this were the case, i.e. if the underlying process generating the sequences satisfies the Markov property, the learning problem would be reduced to learning a mapping from each step to the next. Instead, steps in the sequence may depend on some or all of the preceding steps in the sequence. Therefore, a main challenge faced by a sequence learning model is to capture relevant information from the part of the sequence seen so far. This ability to capture relevant information about future sequences it may receive must be developed during training; the network must learn the ability to build up internal representations which encode relevant aspects of the sequence that is received.
1.3 INCREMENTAL SEQUENCE LEARNING
The dependency on the partial sequence received so far provides a special opportunity for incremental learning that is specific to sequence learning. Whereas the examples in a supervised learning problem bear no known relation to each other, the steps in a sequence have a very specific relation; later steps in the sequence can only be learned well once the network has learned to develop the appropriate internal state summarizing the part of the sequence seen so far. This observation leads to the idea that sequence learning may be expedited by learning to predict the first few steps in each sequence first and, once reasonable performance has been achieved and (hence) a suitable internal representation of the initial part of the sequences has been developed, gradually increasing the length of the partial sequences used for training.
A prefix of a sequence is a consecutive subsequence (a substring) of the sequence starting from the first element; e.g. the prefix S3 of a sequence S consists of the first 3 steps of S. We define Incremental Sequence Learning as an approach to sequence learning whereby learning starts out by using only a short prefix of each sequence for training, and where the length of the prefixes used for training is gradually increased, up to the point where the complete sequences are used. The structure of sequence learning problems suggests that adequate modeling of the preceding part of the sequence
is a requirement for learning later parts of the sequence; Incremental Sequence Learning draws the consequence of this by learning to predict the earlier parts of the sequences first.
1.4 RELATED WORK
In presenting the framework of Curriculum Learning, Bengio et al. (2009) provide an example within the domain of sequence learning, more specifically concerning language modeling. There, the vocabulary used for training on word sequences is gradually increased, i.e. the subset of sequences used for training is gradually increased; this is analogous to one of the comparison methods used here. Another specialization of Curriculum Learning to the context of sequence learning described by Bengio et al. (2015) addresses the discrepancy between training, where the true previous step is presented as input, and inference, where the previous output from the network is used as input; with scheduled sampling, the probability of using the network output as input is adapted to gradually increase over time. Zaremba and Sutskever (2014) apply curriculum learning in a sequence-tosequence learning context where a neural network learns to predict the outcome of Python programs. The generation of programs forming the training data is parameterized by two factors that control the complexity of the programs: the number of digits of the numbers used in the programs and the degree of nesting. While a number of different instantiations of incremental or curriculum learning have been described in the context of sequence learning, no clear guidance is available on which forms are effective. The particular form explored here of learning to predict the earlier parts of sequences first is straightforward, it makes use of the particular structure of sequence learning problems, and it is easy to implement; yet it has received very limited attention so far.
2 MNIST HANDWRITTEN DIGITS AS PEN STROKE SEQUENCES
2.1 MOTIVATION FOR REPRESENTING DIGITS AS PEN STROKE SEQUENCES
The classification of MNIST digit images, see LeCun and Cortes (2010), is one example of a task on which the success of deep learning has been demonstrated convincingly; a test error rate of 0.23% was obtained by Ciresan et al. (2012) using Multi-column Deep Neural Networks. To obtain a sequence learning data set for evaluating Incremental Sequence Learning, we created a variant of the familiar MNIST handwritten digit data set provided by LeCun and Cortes (2010) where each digit image is transformed into a sequence of pen strokes that could have generated the digit.
One motivation for representing digits as strokes is the notion that when humans try to discern digits or letters that are difficult to read, it appears natural to trace the line so as to reconstruct what path the author’s pen may have taken. Indeed, Hinton and Nair (2005) note that the idea that patterns can be recognized by figuring out how they were generated was already introduced in the 1950’s, and describe a generative model for handwritten digits that uses two pairs of opposing springs whose stiffnesses are controlled by a motor program.
Pen stroke sequences also form a natural and efficient representation for digits; handwriting constitutes a canonical manifestation of the manifold hypothesis, according to which “real-world data presented in high dimensional spaces are expected to concentrate in the vicinity of a manifoldM of much lower dimensionality dM, embedded in high dimensional input space Rdx”; see Bengio et al. (2013). Specifically: (i) the vast majority of the pixels are white, (ii) almost all digit images consist of a single connected set of pixels, and (iii) the shapes mostly consist of smooth curved lines. This suggests that collections of pen strokes form a natural representation for the purpose of recognizing digits.
The relevance of the manifold hypothesis can also be appreciated by considering the space of all 2-D 28x28 binary pixel images; when sampling uniformly from this space, one is likely to only encounter images resembling TV noise, and the chances of observing any of the 70000 MNIST digit images is astronomically small. By contrast, a randomly generated pen stroke sequence is not unlikely to resemble a part of a digit, such as a short straight or curved line segment. This increased alignment of the digit data with its representation in the form of pen stroke sequences implies that the amount of computation required to address the learning problem can potentially be vastly reduced.
2.2 CONSTRUCTION OF THE PEN STROKE SEQUENCE DATA SET
The MNIST handwritten digit data set consists of 60000 training images and 10000 test images, each forming 28 x 28 bit map images of written numerical digits from 0 to 9. The digits are transformed into one or more pen strokes, each consisting of a sequence of pen offset pairs (dx, dy). To extract the pen stroke sequences, the following steps are performed:
1. Incremental thesholding. Starting from the original MNIST grayscale image, the following characteristics are measured:
• The number of nonzero pixels • The number of connected components, for both the 4-connected and 8-connected
variants.
Starting from a thresholding level of zero, the thresholding level is increased stepwise, until either (A) the number of 4-connected or 8-connected components changes, (B) the number of remaining pixels drops below 50% of the original number, or (C) the thresholding level reaches a preselected maximum level (250). When any of these conditions occur, the previous level (i.e. the highest thresholding level for which none of these conditions occurred) is selected.
2. A common method for image thinning, described by Zhang and Suen (1984), is applied.
3. After the thresholding and thinning steps, the result is a skeleton of the original digit image that mostly consists of single-pixel-width lines.
4. Finding a pen stroke sequence that could have produced the digit skeleton can be viewed as a Traveling Salesman Problem where, starting from the origin, all points of the digit skeleton are visited. Each point is represented by the pen offset (dx, dy) from the previous to the current point. For any transition to a non-neighboring pixel (based on 8-connected distance), an extra step is inserted with (dx, dy) = (0, 0) and with eos = 1 (end-of-stroke), to indicate that the current stroke has ended and the pen is to be lifted off the paper. At the end of each sequence, a final step with values (0, 0, 1, 1) is appended. The fourth value represents eod, end-of-digit. This final tuple of the sequence marks that both the current stroke and the current sequence have ended, and forms a signal that the next input presented to the network will belong to another digit.
It is important to note that the thinning operation discards pixels and therefore information; this implies that the sequence learning problem constructed here should be viewed as a new learning problem, i.e. performance on this new task cannot be directly compared to results on the original MNIST classification task. While for many images the thinned skeleton is an adequate representation that retains original shape, in other cases relevant information is lost as part of the thinning process.
3 NETWORK ARCHITECTURE
We adopt the approach to generative neural networks described by Graves (2013) which makes use of mixture density networks, introduced by Bishop (1994). One sequence corresponds to one complete image of a digit skeleton, represented as a sequence of 〈dx, dy, eos, eod〉 tuples, and may contain one or more strokes; see previous section.
The network has four input units, corresponding to these four input variables. To produce the input for the network, the (dx, dy) pairs are scaled to yield two real-valued input variables dx and dy. The
variables indicating the end-of-stroke (EOS) and end-of-digit (EOD) are binary inputs. Two hidden LSTM layers, see Hochreiter and Schmidhuber (1997), of 200 units each are used.
The input units receive one step of a sequence at a time, starting with the first step. The goal for the output units is to predict the immediate next step in the sequence, but rather than trying to directly predict dx and dy, the output units represent a mixture of bivariate Gaussians. The output layer consists of the end of stroke signal (EOS), and a set of means µi , standard deviations σi, correlations ρi, and mixture weights πi for each of theM mixture components, where the number of mixture componentsM = 17 was found empirically to yield good results and is used in the experiments presented here. Additionally, a binary indicator signaling the end of digit (EOD) is used, to mark the end of each sequence. In addition to these output elements for predicting the pen stroke sequences, 10 binary class variable outputs are added, representing the 10 digit classes. This facilitates switching the task from sequence prediction to sequence classification, as will be discussed later; the output of these units is ignored in the sequence prediction experiments. The number of output units depends on the number of mixture components used, and equals 6M+ 2 + 10 = 114. For regularization, we found in early experiments that using the maximum weight as a regularization term produced better results than using the more common L-2 regularization. This approach can be viewed as L-∞-norm regularization, and has been used previously in the context of regularization, see e.g. Schmidt et al. (2008).
The definition of the sequence prediction loss LP follows Graves (2013), with the difference that terms for the eod and for the L-∞ loss are included:
L(x) = T∑
t=1
−log ∑ j πjtN (xt+1|µ j t , σ j t , ρ j t )  − { log eost if (xt+1)3 = 1 log (1− eost) otherwise
− {
log eodt if (xt+1)4 = 1
log (1− eodt) otherwise + λ||w||∞
4 INCREMENTAL SEQUENCE LEARNING AND COMPARISON METHODS
Below we describe Incremental Sequence Learning and three comparison methods, where two of the comparison methods are other instantiations of curriculum learning, and the third comparison is regular sequence learning without a curriculum learning aspect.
• Regular sequence learning The baseline method is regular sequence learning; here, all training data is used from the outset.
• Incremental Sequence Learning: increasing sequence length Predicting the second step of a sequence given the first step is a straightforward mapping problem that can be handled using regular supervised learning methods. The prediction of later steps in the sequence can potentially depend on all preceding steps, and for some cases may only be learned once an effective internal representation has been developed that summarizes relevant information present in the preceding part of the sequence. For predicting the 17th step for example, the available input consist of the previous 16 steps, and the network must learn to construct a compact representation of the preceding steps that have been seen. More specifically, it must be able to distinguish between subspaces of the sequence space that correspond to different distributions for the next step in the sequence. The number of possible contexts grows exponentially with the position in the sequence, and the task of summarizing the preceding sequence therefore potentially becomes more difficult as a function of the position within the sequence. The problem of learning to predict steps later on in the sequence is therefore potentially much harder than learning to predict the earlier steps. In Incremental Sequence Learning therefore, the length of sequences presented to the network is increased as learning progresses.
• Increasing training set size Bengio et al. (2009) describe an application of curriculum learning to sequence learning, where the task is to predict the best word which can follow a given context of words in a correct English sentence. The curriculum strategy used there is to grow the vocabulary size. Transferring this to the context of pen stroke sequence generation, the most straightforward translation is to use subsets of the training data that grow in size, where the order of examples that are added to the training set is random.
• Increasing number of classes The network is first presented with sequences from only one digit class; e.g. all zeros. The number of classes is increased until all 10 digit classes are represented in the training data.
All three curriculum learning methods employ a threshold criterion based on the training RMSE; once a specified level of the RMSE has been reached, the set of training examples (determined by the number of sequence steps used, the number of sequences used, or the number of digits) is increased. We note that many possible variants of this simple adaptive scheme are possible, some of which may provide improvements of the results.
5 EXPERIMENTAL SETTINGS
In this section, we describe the experimental setup in detail.
The configuration of the baseline method, regular sequence learning, is as follows. The number of mixture componentsM = 17, two hidden layers of size 200 are used. A batch size of 50 sequences per batch is used in these first experiments. The learning rate is α = 0.0025, with a decay rate of 0.99995 per epoch. The order of training sequences (not steps within the sequences) is randomized. The weight of the regularization component λ = 0.25. In these first experiments, a subset of 10 000 training sequences and 5 000 test sequences is used. The error measure in these figures is the RMSE of the pen offsets (unscaled) predicted by the network given the previous pen movement.
The RMSE is calculated based on the difference between the predicted and actual (dx, dy) pairs, scaled back to their original range of pixel units, so as to obtain an interpretable error; the eos and eod components of the error, which do form part of the loss, are not used in this error measure. For the method where the sequence length is varied, the number of individual points (input-target pairs) that must be processed per sequence varies over the course of a run. The number of sequences processed (or collections thereof such as batches or epochs) is therefore no longer an adequate measure of computational expense; performance is therefore reported as a function of the number of points processed.
Details per method:
• Incremental Sequence Learning The initial sequence length is 2, meaning that the first two points of each sequence are used, i.e. after feeding the first point as input, the second point is to be predicted. Once the training RMSE drops below the threshold value of 4, the length is doubled, up to the point where it reaches the maximum sequence length.
• Increasing training set size The initial training set size is 10. Each time the RMSE threshold of 4 is reached, this amount is doubled, up to the point where the complete set of training sequences is used.
• Increasing number of digit classes The initial number of classes is 1, meaning that only sequences representing the first digit (zero) are used. Each time the RMSE threshold of 4 is reached, this amount is doubled, up to the point where all 10 digit classes are used.
6 EXPERIMENTAL RESULTS
6.1 SEQUENCE PREDICTION: COMPARISON OF THE METHODS
Figures 5 shows a comparison of the results of the four methods. The baseline method (in red) does not use curriculum learning, and is presented with the entire training set from the start. Incremental Sequence Learning (in green) performs markedly better than all comparison methods. It reaches the best test performance of the baseline methods twenty times faster; see the horizontal dotted black line. Moreover, Incremental Sequence Learning greatly improves generalization; on this subset of the data, the average test performance over 10 runs reaches 1.5 for Incremental Sequence Learning vs 3.9 for regular sequence learning, representing a reduction of the error of 74%.
We furthermore note that the variance of the test error is substantially lower than for each of the other methods, as seen in the performance graphs; and where the three comparison methods reach their best test error just before 4 · 106 processed sequence steps and then begin to deteriorate, the test error for incremental sequence learning continues to steadily decrease over the course of the run.
The two other curriculum methods do not provide any speedup or advantage compared to the baseline method, and in fact result in a higher test error; indiscriminate application of the curriculum learning principle apparently does not guarantee improved results and it is important therefore to discover which forms of curriculum learning can confer an advantage.
To explain the dramatic improvement achieved by Incremental Sequence Learning, we consider two possible hypotheses: H1: The number of sequences per batch is fixed (50), but the number of sequence steps or points varies, and is initially much smaller (2) for Incremental Sequence Learning. Thus, when measured in terms of the number of points that are being processed, the batch size for Incremental Sequence Learning is initially much smaller than for the remaining methods, and it increases adaptively over time. HypothesisH1 therefore is that (A) the smaller batch size improves performance, see Keskar et al. (2016) for earlier findings in this direction, and/or (B) the adaptive batch size aspect has a positive effect on performance. H2: Effectively learning later parts of the sequence requires an adequate internal representation of the preceding part of the sequence, which must be learned first; this formed the motivation for the Incremental Sequence Learning method.
To test the first hypothesis, H1, we design a second experiment where the batch size is no longer defined in terms of the number of sequences, but in terms of the number of points or sequence steps, where the number of points is chosen such that the expected total number of points for the baseline method remains the same. Thus, whereas a batch for regular sequence learning contains 50 sequences of length 40 on average yielding 2000 points, Incremental Sequence Learning will start out with batches containing 1000 sequences of 2 points each, yielding the same total number of points.
Figure 6 shows the results. This change reduces the speedup during the earlier part of the runs, and thus partially explains the improvements observed with Incremental Sequence Learning. However, part of the speedup is still present, and moreover the three other observed improvements remain:
• Incremental Sequence Learning still features strongly improved generalization performance • Incremental Sequence Learning still has a much lower variance of the test error • Incremental Sequence Learning still continues improving at the point where the test perfor-
mance of all other methods starts deteriorating
In summary, the adaptive and initially smaller batch size of Incremental Sequence Learning explains part of the observed improvements, but not all. We therefore test to what extent hypothesisH2 plays a role. To see whether the ability to first learn a suitable representation based on the earlier parts of the sequences plays a role, we compare the situation where this effect is ruled out. A straightforward way to achieve this is to use Feed-Forward Neural Networks (FFNNs); whereas Recurrent Neural Networks (RNNs) are able to learn such a representation by learning to build up relevant internal state, FFNNs lack this ability. Therefore if any advantage of Incremental Sequence Learning is seen when using FFNNs, it cannot be due to hypothesis H2. Conversely, if using FFNNs removes the advantage, the advantage must have be due to the difference between FFNNs and RNNs, which exactly corresponds to the ability to build up an informative internal representation, i.e.H2. Since
we want to explain the remaining part of the effect, we also use a batch size based on the number of points, as in Experiment 2.
Figure 7 shows the results. As the figure shows, when using FFNNs, the advantage of Incremental Sequence Learning is entirely lost. This provides a clear demonstration that both of the hypothesesH1 andH2 play a role. Together the two hypotheses explain the total effect of the difference, suggesting that the proposed hypotheses are also the only explanatory factors that play a role.
It is interesting to compare the performance of the RNN and their FFNN variants, by comparing the results of Experiments 2 and 3. From this comparison, it is seen that for Incremental Sequence Learning, the RNN variant achieves improved performance compared to the FFNN variant, as would be expected, since a FFNN cannot make use of any knowledge of the preceding part of the sequence and is thus limited to learning a general mapping between two subsequent pen offsets pairs (dxk, dyk) and (dxk+1, dyk+1). However, it is the only method of the four to do so; for all three other methods, around the point where test performance for the RNN variants starts to deteriorate (after around 4 · 106 processed sequence steps), FFNN performance continues to improve and surpasses that of the RNN variants. This suggests that Incremental Sequence Learning is the only method that is able to utilize information about the preceding part of the sequence, and thereby surpass FFNN performance. In terms of absolute performance, a strong further improvement can be obtained by using the entire training set, as will be seen in the next section. These results suggest that learning the earlier parts of the sequence first can be instrumental in sequence learning.
6.2 LOSS AS A FUNCTION OF SEQUENCE POSITION
To further analyze why variation of the sequence length has a particularly strong effect on sequence learning, we evaluate how the relative difficulty of learning a sequence step relates to the position within the sequence. To do so, we measure the average loss contribution of the points or steps within a sequence as a function of their position within the sequence, as obtained with a learning method that
learns entire sequences (no incremental learning), averaged over the first hundred epochs of training. Figure 8 shows the results.
The first steps are fundamentally unpredictable as the network cannot know which example it will receive next; accordingly, at the start of the sequence, the error is high, as the method cannot know in advance what the shape or digit class of the new sequence will be. Once the first steps of the sequence have been received and the context increasingly narrows down the possibilities, the loss for the prediction of the next steps steeply drops. Subsequently however, as the position in the sequence advances, the loss increases strongly, and exceeds the initial uncertainty of the first steps. This effect may be explained by the fact that the number of possible preceding contexts increases exponentially, thus posing stronger requirements on the learning system for steps later on in the sequence.
6.3 RESULTS ON THE FULL MNIST PEN STROKE SEQUENCE DATA SET
The results reported so far were based on a subset of 10000 training sequences and 5000 test sequences, in order to complete a sufficient number of runs for each of the experiments within a reasonable amount of time. Given the positive results obtained with Incremental Sequence Learning, we now train this method on the full MNIST Pen Stroke Sequence Data Set, consisting of 60000 training sequences and 10000 test sequences (Experiment 4). In these experiments, a batch size of 500 sequences instead of 50 is used.
Figure 9 shows the results. Compared to the performance of the above experiments, a strong improvement is obtained by training on this larger set of examples; whereas the best test error in the results above was slightly above 1.5, the test performance for this experiment drops below one; a test error of 0.972 on the full test data set is obtained. An strking finding is that while initially the test error is much larger than the train error, the test error continues to improve for a long time, and approaches the training error very closely; in other words, no overtraining is observed even for relatively long runs where the training performance appears to be nearly converged.
6.4 TRANSFER LEARNING
The first task considered here was to perform sequence learning: predicting step t+1 of a sequence given step t. To adequately perform this task, the network must learn to detect which digit it is being fed; the initial part of a sequence representing a 2 or 3 for example is very similar, but as evidence is growing that the current sequence represents a 3, that information is vital in predicting how the stroke will continue.
Given that the network is expected to have built up some representation of what digit it is reading, an interesting test is to see whether it is able to switch to the task of sequence classification. The input presentation remains the same: at every time step, the recurrent neural network is fed one step of the sequence of pen movements representing the strokes of a digit. However, we now also read the output of the 10 binary class variable outputs. The target for these is a one-hot representation of the digit, i.e. the target value for the output corresponding to the digit is one, and all nine other target values are zero. To obtain the output, softmax is used, and the sequence classification loss LC for the classification outputs is the cross entropy, weighted by a factor γ = 10:
LC = γ ( − 1 N N∑ n=1 [ynlogŷn + (1− yn)log(1− ŷn)] )
In the following experiments, the loss consists of the sequence classification loss LC , to which optionally the earlier sequence prediction loss LP is added, regulated by a binary parameter β:
L = LC + βLP
The network is asked for a prediction of the digit class after each step it receives. Clearly, accurate classification is impossible during the first part of a sequence; before the first point is received, the sequence could represent any of the 10 digits with equal probability. As the sequence is received step by step however, the network receives more information. The prediction produced after receiving the one-but-last step of the sequence, i.e. at the point where the network was previously asked to predict the last step, is used as its final answer for predicting the digit class.
We compare the following variants:
• Transfer learning: sequence classification and sequence prediction Starting from a trained sequence prediction model as obtained in Experiment 4, the earlier loss function is augmented with the sequence classification loss: L = LC + LP
• Transfer Learning: sequence classification only Starting from a trained sequence prediction model, the loss function is switched such that it only reflects the classification performance, and no longer tracks the sequence prediction performance: L = LC
• Learning from scratch, sequence classification and sequence prediction In this variant, learning starts from scratch, and both classification loss and prediction loss are used, as in the first experiment: L = LC + LP
• Learning from scratch, sequence classification only L = LC
Figure 10 shows the results; indeed the network is able to build further on its ability to predict pen stroke sequences, and learns the sequence classification task faster and more accurately than an identical network that learns the sequence classification task from scratch; in this first and straightforward transfer learning experiment based on the MNIST stroke sequence data set, a classification accuracy of 96.0% is reached1. We note that performance on the MNIST sequence data cannot be compared to results obtained with the original MNIST data set, as the information in the input data is vastly reduced. This result sets a first baseline for the MNIST stroke sequence data set; we expect there is ample room for improvement. Simultaneously learning sequence prediction and sequence classification does not appear to provide an advantage, neither for transfer learning nor for learning from scratch.
1This performance was reached after training for 7 · 107 sequence steps, i.e. roughly twice as long as the run shown in the chart
7 GENERATIVE RESULTS
To gain insight into what the network has learned, in this section we report examples of output of the network.
7.1 DEVELOPMENT DURING TRAINING
During training, the network receives each sequence step by step, and after each step, it outputs its expectation of the offset of the next point. In these figures and movies, we visualize the predictions of the network for a given sequence at different stages of the training process. All results have been obtained from a single run of Incremental Sequence Learning.
7.2 UNGUIDED OUTPUT GENERATION, A.K.A. NEURAL NETWORK HALLUCINATION
After training, the trained network can be used to generate output independently. The guidance that is present during training in the form of receiving each next step of the sequence following a prediction is not available here. Instead, the output produced by the network is fed back into the network as its next input, see Figures 12 and 13. Figure 14 shows example results.
7.3 SEQUENCE CLASSIFICATION
The third analysis of the behavior the trained network is to view what happens during sequence classification. At each step of the sequence, we monitor the ten class outputs and visualize their output. As more steps of the sequence are being received, the network receives more information, and adjusts its expectation of what digit class the sequence represents.
●
●
MNIST stroke sequence test image 25
●
● ●
●
● ● ●
● ● ●
● ●
● ● ● ●
● ● ●
● ● ● ●
● ●
●
● ●
● ●
●
●●●
●
5 10 15 20 25 30 35
0 2
4 6
8
Classification output
Sequence step
D ig
it cl
as s
Classification output for a sequence representing a 0. Initially, as the downward part of the curved stroke is being received, the network believes the sequences represents a 4. After passing the lowest point of the figure, it assigns higher likelihood to a 6. Only at the very end, just in time before the sequence ends, the prediction of the network switches for the last time, and a high probability is assigned to the correct class.
● ●
MNIST stroke sequence test image 18
● ●
● ●
●
● ●
● ●
●
● ● ●
● ● ● ● ● ● ●
● ● ●
●
● ● ● ●●
●●●●●●●●●
● ●
●
● ●
●●●
10 20 30 40 50
0 2
4 6
8
Classification output
Sequence step
D ig
it cl
as s
Classification output for a sequence representing a 3. Initially, the networks estimates the sequence to represent a 7. Next, it expects a 2 is more likely. After 20 points have been received, it concludes (correctly) that the sequences represents a 3.
●●
MNIST stroke sequence test image 62
●●
●●●●● ● ●
● ● ● ●
● ● ● ● ●
● ● ● ●
●
● ●
●
● ● ● ● ● ● ● ●
5 10 15 20 25 30 35
0 2
4 6
8
Classification output
Sequence step
D ig
it cl
as s
Classification output for a sequence representing a 9. While receiving the sequence, the dominant prediction of the network is that the sequence represents a five; the open loop of the 9 and the straight top line may contribute to this. When the last points are received, the network consider a 9 to be more likely, but some ambiguity remains.
8 CONCLUSIONS
There are many possible ways to apply the principles of incremental or curriculum learning to sequence learning, but so far a general understanding of which forms of curriculum sequence learning have a positive effect is missing. We have investigated a particular approach to sequence learning where the training data is initially limited to the first few steps of each sequence. Gradually, as the
network learns to predict the early parts of the sequences, the length of the part of the sequences used for training is increased. We name this approach Incremental Sequence Learning, and find that it strongly improves sequence learning performance. Two other forms of curriculum sequence learning used for comparison did not display improvements compared to regular sequence learning. The origins of this performance improvement are analyzed in comparison experiments, as detailed below.
A first observation was that with Incremental Sequence Learning, the time required to attain the best test performance level of regular sequence learning was much lower; on average, the method reached this level twenty times faster, thus achieving a significant speedup and reduction of the computational cost of sequence learning. More importantly, Incremental Sequence Learning was found to reduce the test error of regular sequence learning by 74%.
To analyze the cause of the observed speedup and performance improvements, we first increase the number of sequences per batch for Incremental Sequence Learning, so that all methods use the same number of sequence steps per batch. This reduced the speedup, but the improvement of the generalization performance was maintained. We then replaced the RNN layers with feed forward network layers, so that the networks can no longer maintain information about the earlier part of the sequences. This completely removed the remaining advantage. This provides clear evidence that the improvement in generalization performance is due to the specific ability of an RNN to build up internal representations of the sequences it receives, and that the ability to develop these representations is aided by training on the early parts of sequences first.
Next, we trained Incremental Sequence Learning on the full MNIST stroke sequence data set, and found that the use of this larger training set further improves sequence prediction performance. The trained model was then used as a starting point for transfer learning, where the task was switched from sequence prediction to sequence classification.
We conclude that Incremental Sequence Learning provides a simple and easily applicable approach to sequence learning that was found to produce large improvements in both computation time and generalization performance. The dependency of later steps in a sequence on the preceding steps is characteristic of virtually all sequence learning problems. We therefore expect that this approach can yield improvements for sequence learning applications in general, and recommend its usage, given that exclusively positive results were obtained with the approach so far.
9 RESOURCES
The Tensorflow implementation that was used to perform these experiments is available here: https: //github.com/edwin-de-jong/incremental-sequence-learning
The MNIST stroke sequence data set is available for download here: https: //github.com/edwin-de-jong/mnist-digits-stroke-sequence-data/wiki/ MNIST-digits-stroke-sequence-data
The code for transforming the MNIST digit data set to a pen stroke sequence data set has also been made available: https://github. com/edwin-de-jong/mnist-digits-as-stroke-sequences/wiki/ MNIST-digits-as-stroke-sequences-(code)
ACKNOWLEDGMENTS
The author would like to thank Max Welling, Dick de Ridder and Michiel de Jong for valuable comments and suggestions on earlier versions.

First up, I want to point out that this paper is really long. Like 17 pages long -- without any supplementary material. While ICLR does not have an official page limit, it would be nice if authors put themselves in the reviewer's shoes and did not take undue advantage of this rule. Having 1 or 2 pages in addition to the conventional 8 page limit is ok, but more than doubling the pages is quite unfair.
Now for the review: The paper proposes a new artificial dataset for sequence learning. I call it artificial because it was artificially generated from the original MNIST dataset which is a smallish dataset of real images of handwritten digits. In addition to the dataset, the authors propose to train recurrent networks using a schedule over the length of the sequence, which they call "incremental learning". The experiments show that their proposed schedule is better than not having any schedule on this data set. Furthermore, they also show that their proposed schedule is better than a few other intuitive schedules. The authors verify this by doing some ablation studies over the model on the proposed dataset.
I have following issues with this paper:
-- I did not find anything novel in this paper. The proposed incremental learning schedule is nothing new and is a natural thing to try when learning sequences. Similar idea have already been tried by a number of authors, including Bengio 2015, and Ranzato 2015. The only new piece of work is the ablation studies which the authors conduct to tease out and verify that indeed the improvement in performance is due to the curriculum used.
-- Furthermore, the authors only test their hypothesis on a single dataset which they propose and is artificially generated. Why not use it on a real sequential dataset, such as, language modeling. Does the technique not work in that scenario? In fact I am quite positive that for language modeling where the vocabulary size is huge, the performance gains will be no where close to the 74% reported in the paper.
-- I'm not convinced about the value of having this artificial dataset. Already there are so many real world sequential dataset available, including in text, speech, finance and other areas. What exactly does this dataset bring to the table is not super clear to me. While having another dataset may not be a bad thing in itself, I almost felt that this dataset was created for the sole purpose of making the proposed ideas work. It would have been so much better had the authors shown experiments on other datasets.
-- As I said, the paper is way too long. A significant part of the length of the paper is due to a collection of experiments which are completely un-related to the main message of the paper. For instance, the experiment in Section 6.2 is completely unrelated to the story of the paper. Same is true with the transfer learning experiments of Section 6.4.


CONVOLUTIONAL NEURAL NETWORKS
Several recent works have empirically observed that Convolutional Neural Nets (CNNs) are (approximately) invertible. To understand this approximate invertibility phenomenon and how to leverage it more effectively, we focus on a theoretical explanation and develop a mathematical model of sparse signal recovery that is consistent with CNNs with random weights. We give an exact connection to a particular model of model-based compressive sensing (and its recovery algorithms) and random-weight CNNs. We show empirically that several learned networks are consistent with our mathematical analysis and then demonstrate that with such a simple theoretical framework, we can obtain reasonable reconstruction results on real images. We also discuss gaps between our model assumptions and the CNN trained for classification in practical scenarios.
1 INTRODUCTION
Deep learning has achieved remarkable success in many technological areas (Bengio et al., 2013; Schmidhuber, 2015), including computer vision (Krizhevsky et al., 2012; Szegedy et al., 2015; Simonyan and Zisserman, 2015), automatic speech recognition (Hinton et al., 2012; Hannun et al., 2014), natural language processing (Collobert et al., 2011; Mikolov et al., 2013; Cho et al., 2014), bioinformatics (Chicco et al., 2014), even high energy particle physics (Baldi et al., 2014). In particular, deep Convolutional Neural Networks (CNNs) (LeCun et al., 1989; Krizhevsky et al., 2012; Simonyan and Zisserman, 2015) have been a critical enabling technique for analyzing images and sequential data.
Following the unprecedented success of deep networks, there has been some theoretical work (e.g., Arora et al. (2014; 2015); Paul and Venkatasubramanian (2014)) that suggest several mathematical models for different deep learning architectures. However, theoretical analysis and understanding lag behind the very rapid evolution and empirical success of deep architectures, and more theoretical analysis is needed to better understand the state-of-the-art deep architectures, and possibly to improve them further.
In this paper, we attempt to address the gap between the empirical success and theoretical understanding of the Convolutional Neural Nets, in particular its invertibility (i.e., reconstructing the input from the hidden activations), by analyzing a simplified mathematical model using random weights.1
This property is intriguing because convolutional neural networks are typically trained with discriminative objectives (i.e., unrelated to reconstruction) with a large amount of labels, such as the ImageNet dataset. For example, Dosovitskiy and Brox (2016) used upsampling-deconvolutional architectures to invert the hidden activations of feedforward CNNs to the input domain. In other related work, Zhao et al. (2016) proposed stacked a what-where network via a (deconvolutional) decoder and demonstrate its promise in unsupervised and semi-supervised settings. Bruna et al. (2014) studied signal discovery from generalized pooling operators using image patches on non-convolutional small scale networks and datasets. Zhang et al. (2016) showed that CNNs discriminately trained for image classification (e.g., VGG Net (Simonyan and Zisserman, 2015)) are almost fully invertible using pooling switches. Despite these interesting results, there is no clear theoretical explanation as to why CNNs are invertible yet.
We introduce three new concepts that, coupled with the accepted notion that images have sparse representations, guide our understanding of CNNs:
1For more discussion about random filters, see Sections 2.1 and 4.1.
1. we provide a particular model of sparse linear combinations of the learned filters that are consistent with natural images; also, this model of sparsity is itself consistent with the feedforward network;
2. we show that the effective matrices that capture explicitly the convolution of multiple filters exhibit a model-Restricted Isometry Property (model-RIP) (Baraniuk et al., 2010); and
3. our model can explain each layer of the feedforward CNN algorithm as one iteration of Iterative Hard Thresholding (IHT) (Blumensath and Davies, 2009) for model-based compressive sensing and, hence, we can reconstruct the input simply and accurately.
In other words, we give a theoretical connection to a particular model of model-based compressive sensing (and its recovery algorithms) and CNNs. We show empirically that large-scale deep convolution networks are consistent with our mathematical analysis. We then demonstrate that with such a simple theoretical framework, we can obtain reasonable reconstruction results on real images, using filters from trained networks. Finally, we observe that it makes a significant difference which filters one uses for encoding and decoding, whether they are trained specifically for reconstruction, or random, or the same for both procedures. This paper explores these properties and elucidate specific empirical aspects that any more sophisticated mathematical model should take into account.2
2 PRELIMINARIES
In this section, we set the stage for our mathematical analysis in Section 3. We begin with discussion on the use of random weights in (convolutional) neural networks, and then provide the definitions and models for CNNs. Then, we discuss compressive sensing and sparse signal recovery. We define a particular model of sparsity that we will use throughout our analysis and detail the Iterative Hard Thresholding (IHT) algorithm which is the basis of our reconstruction analysis.
In order to simplify our notation and to make clear our analysis, we focus on a single layer in the analysis instead of multiple layers.3 Also, we assume that all of our input signals are vectors rather than matrices and that any operations we would ordinarily carry out on images (e.g., convolving with a filter bank, dividing into regions over which we pool coefficients), we do on vectors with the appropriate modifications for a simplified structure. While these assumptions ease our exposition, they do not change the nature of our arguments nor their implications for images. Furthermore, we demonstrate the validity of our results in two-dimensional natural images.
2.1 EFFECTIVENESS OF GAUSSIAN RANDOM FILTERS
We analyze theoretically CNNs with Gaussian random filters, which have been surprisingly effective in unsupervised and supervised deep learning tasks. Jarrett et al. (2009) showed that random filters in 2-layer CNNs work well for image classification. In addition, Saxe et al. (2011) observed that convolutional layer followed by pooling layer is frequency selective and translation invariant, even with random filters, and these properties lead to good performance for object recognition tasks. On the other hand, Giryes et al. (2016) proved that CNNs with random Gaussian filters have metric preservation property, and they argued that the role of training is to select better hyperplanes discriminating classes by distorting boundary points among classes. According to their observation, random filters are in fact a good choice if training data are initially well-separated. Also, He et al. (2016) empirically showed that random weight CNNs can do image reconstruction well.
To better demonstrate the effectiveness of Gaussian random CNNs, we evaluate their classification performance on CIFAR-10; see Section 4.1 for details. We find that a 3-layer Gaussian random CNN is able to achieve ∼ 75% accuracy on the test set, with only the last classifier layer optimized, (see Table 1 for more details). Even though this number is far from the state-of-the-art results, it is surprisingly good considering the networks are almost untrained. Our theoretical results may provide another new perspective on explaining these phenomena.
2We note that our model may not be an exact replica of a real setting, but for mathematical analysis, it is a simplified but representative abstraction of practical settings. A number of works show that random weight CNNs still achieve surprisingly good classification accuracy although they may not match the state-of-the-art results; see Sections 2.1 and 4.1 for more discussion.
3We can extend the equivalency on a single layer of CNNs to multiple layer CNNs simply by using the output on one layer as the input to another, still using the steps of the inner loop of IHT.
2.2 CONVOLUTIONAL NEURAL NETS
We define a single layer of our CNN as follows. We assume that the input signal x consists of M channels, each of length D, and we write x ∈ RMD. For each of the input channels, m = 1, . . . ,M , let wi,m, i = 1, . . . ,K denote one of K filters, each of length `. Let t be the stride length, the number of indices by which we shift each filter. Note that t can be larger than 1.
We assume that the number of shifts, n = (D− `)/t+ 1, is an integer. Let wji,m be a vector of length D that consists of the (i,m)-th filter shifted by jt, j = 0, . . . , n−1 (i.e., wji,m has at most ` non-zero entries). We will concatenate over the M channels each of these vectors (as row vectors) to form a large matrix, W , which is the Kn×MD matrix made up of K blocks of the n shifts of each filter in each of M channels. We assume that Kn ≥MD. We also assume that the Kn row vectors of W span RMD and that we have normalized the rows so that they have unit `2 norm. We assume that the hidden units of the feed-forward CNN are computed by multiplying an input signal x ∈ RMD by the matrix W (i.e., convolving, in each channel, by a filter bank of size K, and summing over the channels to obtain Kn outputs), applying the ReLU function to the Kn outputs, and then selecting the value with maximum absolute value in each of the K blocks; i.e., we perform max pooling over each of the convolved filters and sum over the channels.4 We use h = Wx for the hidden activation computed by a single layer CNN without pooling. Figure 1 illustrates the architecture.
2.3 COMPRESSIVE SENSING
Let Φ be a i×j matrix with j > i. We say that Φ satisfies the Restricted Isometry Property RIP(k, δk) (or, just RIP) if there is a distortion factor δk > 0 such that for all z ∈ Rj with exactly k non-zero entries, (1− δk)‖z‖22 ≤ ‖Φz‖22 ≤ (1 + δk)‖z‖22. If Φ satisfies RIP (for appropriate sparsity level k and sufficiently small δk) and if z ∈ Rj is k-sparse, then, given the vector x = Φz ∈ Ri, we can efficiently recover z (see Candés (2008) for more details)5. There are many efficient algorithms for doing so, including `1 sparse coding (e.g., `2 minimization with `1 regularization) and greedy, iterative algorithms (such as Iterative Hard Thresholding or IHT).
Model-based compressive sensing. While sparse signals are a natural model for some applications, they are less realistic for CNNs. We consider a vector z ∈ RKn as the true sparse code for generating the CNN input x with a particular model of sparsity. Rather than permitting k non-zero entries anywhere in the vector z, we divide the support of z into K contiguous blocks of size n and we stipulate that from each block there is at most one non-zero entry in z with a total of k non-zero
4The convolution can be computed more efficiently than a straight-forward matrix multiplication, but they are mathematically equivalent.
5We note that this is a sufficient condition and that there are other, less restrictive sufficient conditions, as well as more complicated necessary conditions. Furthermore, we have not given the exact, quantitative relations amongst the parameters. For simplicity, we stick with this definition.
entries. We call a vector with this sparsity model model-k-sparse and denote the union of all ksparse subspaces with this structure Mk. It is clear that Mk contains nk ( K k ) subspaces. In our analysis, we consider linear combinations of two model-k-sparse signals. To be precise, suppose that z = α1z1 + α2z2 is the linear combination of two elements inMk. Then, we say that z lies in the linear subspaceM2k that consists of all linear combinations of vectors fromMk. We say that a matrix Φ satisfies the model-RIP condition for parameter k if, there is a distortion factor δk > 0 such that, for all z ∈Mk,
(1− δk)‖z‖22 ≤ ‖Φz‖22 ≤ (1 + δk)‖z‖22. (1)
See Baraniuk et al. (2010) for the definitions of model sparse and model-RIP, as well as the necessary modifications to account for signal noise and compressible (as opposed to exactly sparse) signals (which we have neglected to consider to keep our analysis simple). Intuitively speaking, a matrix that satisfies the model-RIP is a nearly an orthonormal matrix for a particular set of sparse vectors with a particular sparsity model or pattern.
For our analysis, we also need matrices Φ that satisfy the model-RIP condition for vectors z ∈M2k. We denote the distortion factor δ2k for such matrices. Note that δk ≤ δ2k < 1.
Algorithm 1 Model-based IHT Input: model-RIP matrix Φ, measure-
ments x = Φz, structured sparse approximation algorithm M
Output: k-sparse approximation z 1: Initialize z0 = 0, d = x, i = 0 2: while stopping criteria not met do 3: i← i+ 1 4: b← zi−1 + ΦTd 5: zi ←M(b, k) 6: d← x−Φzi 7: end while 8: return z ← zi
Many efficient algorithms have been proposed for sparse coding and compressive sensing (Olshausen et al., 1996; Mallat and Zhang, 1993; Beck and Teboulle, 2009). As with traditional compressive sensing, there are efficient algorithms for recovering model-k-sparse signals from measurements (see Baraniuk et al. (2010)), assuming the existence of an efficient structured sparse approximation algorithm M, that given an input vector and the sparsity parameter, returns the vector closest to the input with the specified sparsity structure.
In convolutional neural networks, the max pooling operator finds the downsampled activations that are closest to the activations of the original size by retaining the most significant values. The max pooling can be viewed as two steps: 1) zeroing out the locally non-maximum values;
2) downsampling the activations with the locally maximum values retained. To study the pooled activations with sparsity structures, we can recover dimension loss from the second step (downsampling step) by an unsampling operator. This procedure defines our structured sparse approximation algorithm M(z, k), where z is the original (unpooled) code, and k is the sparsity parameter for further sparsification, which guarantees that M(z, k) is a model-k-sparse signal. With the standard layered formulation for neural networks, we have
M(z, k) = block-sparsify(upsample(max-pool(z), s), k), (2)
where s denotes the upsampling switches that indicate where to place the non-zero values in the upsampled activations. Taking the pooling switches known from the max pooling operation as s, we specifically define M as the nesting of the max pooling and the unpooling with known switch. We define this special case as
Mknown(z, k) = block-sparsify(upsample(max-pool(z),max-pool-switch(z)), k). (3)
Alternatively, using the fixed uniform switches as s, we specifically define M as the nesting of the max pooling and the naive unsampling, denoted by Mfixed. In the rest of this paper, our theoretical analysis are generic to any type of valid upsampling switches6, so we use M(z, k) to denote the structured sparse approximation algorithm without worrying about s. The two special cases Mknown and Mfixed are used in the empirical analysis when we need to specify M(z, k) as a fully concrete operator.
The main recovery algorithm that we focus on is a model-sparse version of Iterative Hard Thresholding (IHT) (see Blumensath and Davies (2009)), not because we are interested in recovering modelsparse signals, per se, but because one iteration of IHT for our model of sparsity captures exactly
6Valid switches should place a non-zero value at exactly one location.
a feedforward CNN.7 Algorithm 1 describes the model-based IHT algorithm. In particular, the sequence of steps 4–6 in the middle IHT (without the outer iterative loop) is exactly one layer of a feedforward CNN. As a result, the theoretical analysis of IHT for model-based sparse signal recovery serves as a guide for how to analyze the approximation activations of a CNN.
3 ANALYSIS
To motivate our more formal analysis, we begin with a simple example. Suppose that the matrix W is an orthonormal basis for RMD and define Ψ = [ W T −W T ] .
Proposition 1. A one-layer CNN using the matrix ΨT , with no pooling, gives perfect reconstruction (with the matrix Ψ) for any input vector x ∈ RMD.
Proof. Because we have both the positive and the negative dot products of the signal with the basis vectors in ReLU(ΨTx) = ReLU ([
Wx −Wx
]) , we have positive and negative versions of the hidden
units h+ = ReLU(Wx) and h− = ReLU(−Wx) where we decompose h = Wx = h+ − h− into the difference of two non-negative vectors, the positive and the negative entries of h. From this decomposition, we can easily reconstruct the original signal via
Ψ [ h+ h− ] = [ W T −W T ] [h+ h− ] = W T (h+ − h−) = W Th = W TWx = x.
In the example above, we have pairs of vectors (w,−w) in our matrix Ψ. This settings allow us to turn what would ordinarily be a nonlinear function, ReLU, into a linear one. In fact, the assumption that trained CNN filters come in positive and negative is validated by Shang et al. (2016), which makes a CNN much easier to analyze within the model compressed sensing framework.
Suppose that we have a vector z that we split into positive and negative components, z = z+ − z−, and that we synthesize (or construct) a signal x from z using the matrix [ W T −W T ] . Then, we have [ W T −W T ] [z+ z− ] = W T (z+ − z−) = W Tz = x.
Next, suppose that we multiply x = W Tz by the transpose of the same matrix, we find [ W −W ] x =[
WW Tz −WW Tz
] and, if we apply ReLU to this vector, we produce [ (WW Tz)+ (WW Tz)− ] a vector that is split
into its positive and negative components. To determine whether or not we have “reconstructed” the vector z, the structure of the product WW T is crucial. In addition, this calculation shows that if we have both positive and negative pairs of filters or vectors, then the ReLU function applied to both the positive and negative dot products simply splits the vector into the positive and negative components. These components are then reassembled in the next computation. For this reason, in the analysis in the following sections, it is sufficient to consider W Tz = x and Wx = h with max pooling alone applied to h, assuming that all of the entries in the vectors are real numbers, rather than only non-negative.
3.1 MODEL-RIP AND RANDOM FILTERS
Our first main result says that if we use Gaussian random filters in our CNN, then, with high probability, the transpose of the matrix W formed by the convolutions with these filters has the model-RIP property. In other words, Gaussian random filters generate a matrix whose transpose W T is almost an orthonormal transform for sparse signals with a particular sparsity pattern (that is consistent with our pooling procedure). The bounds in the theorem tell us that we must balance the size of the filters ` and the number of channels M against the sparsity of the hidden units k, the number of the filter banks K, the number of shifts n, the distortion parameter δk, and the failure probability . The proof is in Appendix A.
7Multiple iterations of IHT can improve the quality of signal recovery. However, it is rather equivalent to the recurrent version of CNNs and does not fit to the scope of this work.
Theorem 3.1. Assume that we have MK vectors wi,m of length ` in which each entry is a scaled i.i.d. (sub-)Gaussian random variable with mean zero and variance 1 (the scaling factor is 1/ √ M`). Let t be the stride length (where n = (D − `)/t+ 1) and build the structured random matrix W as the weight matrix in a single layer CNN for M -channel input dimension D. If
M`2
D ≥ Ck
δ2k
( log(K) + log(n)− log( ) ) ,
then, with probability 1− , the MD ×Kn matrix W T satisfies the model-RIP for modelMk with parameter δk.
We also note that the same analysis can be applied to the sum of two model-k-sparse signals, with changes in the constants (that we do not track here). Corollary 3.2. Random matrices with the CNN structure have, with high probability, the model-RIP property forM2k.
Other examples of matrices that satisfy model-RIP (both empirically and via a less sophisticated analysis on the dot products between any two columns) include wavelets and localized Fourier bases; both examples that can be easily and efficiently implemented via convolutions in a CNN.
3.2 RECONSTRUCTION BOUNDS
To distinguish the true sparse code z and its reconstruction, we use ẑ = M(h, k) = M(Wx, k) for the reconstruction by CNN. Our next result tells us that if we compute the hidden units h from an input signal x using a weight matrix W whose transpose has the model-RIP and using max pooling over each filter (ẑ), then we can reconstruct (approximately) the input signal x simply by multiplying the hidden units by W . This result bounds the relative error between the approximate reconstruction x̂ and the input as a function of the distortion for the model-RIP. In our analysis, we assume that the input signal x = W Tz is a sparse linear combination of hidden activations, captured approximately by the filters in W . See Appendix B for the detailed proofs. Part of our analysis also shows that the hidden units ẑ are approximately the putative coefficient vector z in the sparse linear representation for the input signal. Theorem 3.3. We assume that W T satisfies theM2k-RIP with constant δk ≤ δ2k < 1. If we use W in a single layer CNN both to compute the hidden units ẑ and to reconstruct the input x from these hidden units as x̂ so that x̂ = W TM(Wx, k), the error in our reconstruction is
‖x̂− x‖2 ≤ 5δ2k
1− δk √ 1 + δ2k√ 1− δ2k ‖x‖2.
Recall that the structured sparsity approximation algorithm M includes the downsampling caused by pooling and an unsampling operator. Theorem 3.3 is applicable to any type of upsampling switches, so our reconstruction bound is generic to the particular design choice on how to recover the activation size in a decoding neural network.
4 EXPERIMENTAL EVIDENCE AND ANALYSIS
In this section, we provide experimental validation of our theoretical model and analysis. We first validate experimentally the relevance of our assumption by examining the effectiveness of random filter CNNs. We then provide an experimental validation of our theoretical analysis on the synthetic 1D case, then we provide experimental results on more realistic scenarios. In particular, we study popular deep neural networks trained for image classification on the ImageNet ILSVRC 2012 dataset (Deng et al., 2009). We calculate empirical model-RIP bounds for W T , showing that they are consistent with theory. Our results are also consistent with a long line of research shows that it is reasonable to model real, natural images as sparse linear combinations over learned dictionaries (e.g., Boureau et al. (2008); Le et al. (2013); Lee et al. (2008); Olshausen et al. (1996); Ranzato et al. (2007); Yang et al. (2010)). In addition, we verify our theoretical bounds for the reconstruction error ‖x−W T ẑ‖2/‖x‖2 on real images. (This is the relative `2 distance between the original image and the reconstruction.) We investigate both randomly sampled filters and empirically learned filters in these experiments. Our implementation is based on the Caffe (Jia et al., 2014) and MatConvNet (Vedaldi and Lenc, 2015) toolboxes.
4.1 EVALUATION OF GAUSSIAN RANDOM CNNS ON CIFAR-10
To show the practical relevance of our theoretical assumptions on using random filters for CNNs as stated in Section 2.1, we evaluate simple CNNs with Gaussian random filters (with i.i.d. zeromean unit-variance entries) on the CIFAR-10 dataset. The goal of this experiment is not to achieve state-of-the-art results, but to examine practical relevance of our assumption on random filter CNNs. Once the CNNs weights are initialized (randomly), they are fixed during the training of the classifiers. Specifically, we test random CNNs with 1, 2, and 3 convolutional layers, where we use ReLU as the activation. A 2×2 max pooling layer follows each convolutional layer to down-sample the feature map.8 We experiment with different filter sizes (3, 5, 7) and numbers of channels (64, 128, 256, 1024, 2048) and report the classification accuracy of the best-performing architectures based on cross-validation in Table 1. We also report the best performance using learnable filters for comparison. More details about the architectures can be found in Section C.1 of the supplementary materials. We observe the CNNs with Gaussian random filters achieve surprisingly good classification performance (implying that they serve as reasonable representation of input data), although fully learnable CNN counterparts perform better. Our experimental results are also consistent with the observations made by Jarrett et al. (2009) and Saxe et al. (2011). Overall, these results seem to suggest that the CNNs with Gaussian random filters might be a reasonable setup which is amenable to mathematical analysis while not being too far off in terms of practical relevance.
4.2 EXPERIMENTAL VALIDATION OF THE ANALYSIS IN 1D SYNTHETIC DATA
We use 1-D synthetic data to empirically show the basic validity of our theory in terms of the modelRIP condition in Equation (1) and reconstruction bound in Theorem 3.3. We plot the histograms of the empirical model-RIP values of 1D Gaussian random filters W ( scaled by 1/ √ lM ) with size l × 1×M ×K = 5× 1× 32× 96 on 1DMk sparse signal z with size D = 32 and sparsity k = 10, whose non-zero elements are drawn from a uniform distribution on [−1, 1]. The histograms in Figure 2a and 2b are tightly centered around 1, suggesting that W T satisfies the model-RIP condition in Equation (1) and its corollary from Lemma B.1 in the supplementary materials. We also empirically show the reconstruction bound in Theorem 3.3 on synthetic vectors x = W Tz (Figure 2c). The reconstruction error is concentrated at around 0.1–0.2 and bound under 0.5. Results in Figure 2 suggests the practical validity of our theory when the model assumptions hold.
4.3 ARCHITECTURES AND DATASET
We conduct the rest of our experimental evaluations on the 16-layer VGGNet (Model D in Simonyan and Zisserman (2015)),9 where the computation is carried out on images; e.g., convolution with a 2-D filter bank and pooling on square regions. In contrast to the theory, the realistic network does not pool activations over all the possible shifts for each filter, but rather on non-overlapping patches. The networks are trained for the large-scale ImageNet classification task, which is important for extending to other supervised tasks in vision. The main findings on VGGNet are presented in the rest of this section; we also provide some analysis on AlexNet (Krizhevsky et al., 2012) in the supplementary materials.
8Implementation detail: We add a batch normalization layer together with a learnable scale and bias before the activation so that we do not need to tune the scale of the filters. The filter weights of the intermediate layers in the CNNs are not trained after random initialization. On top of the network, we use an optional average pooling layer to reduce the feature map size to 4× 4 and a dropout layer for better regularization before feeding the feature to a learnable soft-max classifier for image classification.
9VGGNet is practically important as it is popularly used in the community and is one of the best-performing “single-pathway” networks (i.e., no skip connections). We expect that the ResNet (e.g., trained from ImagNet) can also reconstruct images from its activations well in practice. However, the ResNet architectures are too complicated to be in the scope of our theory without further nontrivial customization.
VGGNet contains five groups of convolution and pooling layers, each group has 2~3 convolutional layers followed by a pooling layer. We denote the j-th convolutional layer in the i-th group “conv(i, j),” and the pooling layer “pool(i).” When we say the activations/features are from i-th layer, we mean they are the output of pool(i). Our analysis is for single convolutional layers. When evaluating the i-th layer, we take the activations from the (i− 1)-th layer, and investigate the filters and output of conv(i, 1).
4.4 2D MODEL-RIP
The key to our reconstruction bound is Theorem 3.3 is the model-RIP condition for our particular model of sparsity in Equation (1). We empirically evaluate the model-RIP property, i.e., ‖W Tz‖/‖z‖, for real CNN filters of the pretrained VGGNet. We use two-dimensional coefficients (or hidden units) z (each block of coefficients is of size D ×D), K filters of size `× `, and pool the coefficients over smaller pooling regions (i.e., not over all possible shifts of each filter). The following experimental evidence suggest that the sparsity model and the model-RIP property of the filters are consistent with what we conclude from the mathematical analysis on the simpler one-dimensional case.
To check the significance of the model-RIP property (i.e., how close ‖W Tz‖/‖z‖ is to 1) in controlled settings, we first synthesize the hidden activations z with sparse uniform random variables, which fully agree with our model assumptions. The sparsity of z is constrained to the average level of the real CNN activations (refer to Table 2). Given the filters of a certain convolutional layer, we use the synthetic z (in equal position to this layer’s output activations) to get statistics for the model-RIP property. To be consistent with succeeding experiments, we choose conv(5, 2), while other layers
show similar results. Figure 3 (a) summarizes the distribution of empirical model-RIP values, which is clearly centered around 1 and satisfies Equation (1) with a short tail roughly bounded by δk < 1. For more details of the algorithm, we normalize the filters from the conv(5, 2) layer, which are `× ` (` = 3). All K = 512 filters with M = 512 input channels are used.10 We set D = 15 (the same as the output activations of conv(5, 2)) and use 2× 2 pooling regions11 (commonly used in recent deep networks). We generate 1000Mk randomly sampled sparse activation (z) maps by first sampling their non-zero supports and then filling elements on the supports uniformly from [−1, 1]. The sparsity is the same as that in conv(5, 1) activations.
Algorithm 2 Sparse hidden activation recovery Input: convolution matrix W , input activation/image x Output: hidden code z, satisfying our model-RIP assump-
tion withMk and reconstructing x with W 1: zinit = argminz ‖x−W Tz‖ 2
2 + λ‖z‖1 2: zmodel = Mknown(zinit, k) 3: z = argminz ‖x−W Tz‖ 2
2 + λ‖z‖1, s.t. zi = 0 if zmodeli = 0 To conduct more realistic experiments, we observe the actual conv(5, 2) activations from VGGNet are not necessarily drawn from a model-sparse uniform distribution. This motivates us to evaluate the empirical model-RIP property on the hidden activations z that reconstruct the actual input activations x from conv(5, 1) by W Tz. Per theory, the x is given by a max pooling layer, so we constrain the sparsity (i.e., the size of the support set is no more than 1 in a pooling region for a single channel). We use a simple and efficient algorithm to recover z from x in Algorithm 2. The algorithm is inspired by “`1 heuristic" method that are commonly used in practice (e.g. Boyd (2015)). As shown in Algorithm 2, we first do `1-regularized least squares without constraining the support set. Max pooling is then applied to figure out the support set for each pooling region. In particular, we use Mknown, defined in (3), to zero out the locally non-maximum values without messing up the support structures. We perform `1-regularized least squares again on the fixed support set to recover the hidden activations satisfying the model sparsity. As shown in Figures 3 (b)–(c), the empirical model-RIP property values for visual activations x from conv(5, 1) with/without ReLU are both close to 1. The center offset to 1 is less than 0.05 and the range bound δk is rough less then 0.05, which agrees with the theoretical bound (1) quite well.
To gain more insight, we summarize the learned filter coherence in Table 4 for all the convolutional layers in VGGNet.12 This measures the correlation or similarity between the columns of WT and is a proxy for the value of the model-RIP parameter δk (which we can only estimate computationally). The smaller the coherence, the smaller δk is, and the better the reconstruction. The coherence of the learned filters is not low, which is inconsistent with our theoretical assumptions. However, the model-RIP property turns out to be robust to this mismatch. It also demonstrates the strong invertibility of CNN in practice.
4.5 RECONSTRUCTION BOUNDS
With model-RIP as a sufficient condition, Theorem 3.3 provides theoretical bounds for layer-wise reconstruction via x̂ = W TM(Wx, k). This operator consists of the projection and reconstruction in one IHT iteration. Without confusion, we refer to it as IHT for notational convenience. We investigate the practical reconstruction errors on Layer 1~4 activations (i.e., pool(1)~(4)) of VGGNet.
To encode and reconstruct intermediate activations of CNNs, we employ IHT with sparsity estimated from the real CNN activations on ILSVRC-2012 validation set (see Table 2). We also reconstruct input images, since CNN inversion is not limited to a single layer, and images are easier to visualize than hidden activations. To implement image reconstruction, we project the reconstructed activations into the image space via a pretrained decoding network as in (Zhang et al., 2016), which extends a similar autoencoder architecture as in (Dosovitskiy and Brox, 2016) to a stacked “what-where” autoencoder (Zhao et al., 2016). The reconstructed activations were scaled to have the same norm as the original activations so that we can feed them into the decoding network.
10We do not remove any filters including those in approximate positive/negative pairs (refer to 3). 11In VGGNet, no pooling layer follows conv(5, 2). Here, we just use it in this way to analyze the convolutionpooling pair targeted by our theory. 12The coherence is defined as the maximum (in absolute value) dot product between distinct pairs of columns of the matrix WT , i.e. µ = maxi 6=j |WiWTj |, where Wi denote the i-th row of matrix W .
As an example, Figure 4 illustrates the image reconstruction results for the hidden activations of the 4-th layer, the ground truth of which is obtained by feeding natural images to the CNNs. Interestingly, the decoding network itself is powerful, since it can reconstruct the glimpse of images with Gaussian random input, as shown in Figure 4 (e). Object shapes are recovered by using the pooling switches only in the “what-where” autoencoder. This result suggests that it is important to determine which pooling units are active and then to estimate these values accurately. These steps are consistent with the steps in the inner loop of any iterative sparse signal reconstruction algorithm.
In Figure 4 (c), we take the pretrained conv(5, 1) filters for IHT. The images recovered from the IHT reconstructed 4-th layer activations are reasonable and the reconstruction quality is significantly better than the random input baseline. We also try Gaussian random filters (Figure 4 (d)), which agree more with the model assumptions (e.g., lower coherence, see Table 4). The learned filters from VGGNet perform equally well visually. IHT ties the encoder and decoder weights (no filter learning for the decoder), so it does not perform as well as the decoding network trained with a huge batch of data (Figure 4 (b)). Nevertheless, we show both theoretically and experimentally decent reconstruction bounds for these simple reconstruction methods on real CNNs. More visualization results for more layers are in the supplementary materials (Figure 5 in Section C.3).
In Table 3, we summarize reconstruction performance for all 4 layers. With random filters, the model assumptions hold and the IHT reconstruction is the best quantitatively. IHT with real CNN filters performs comparable to the best case and much better than the baseline established by the randomly sampled activations.
Additionally, reconstruction performance of IHT is strongly related to the filter coherence, summarized in Table 4. Lower coherence agrees more closely with the model assumptions and leads to higher reconstruction quality. Higher coherence yields worse recovery of the hidden activation (i.e., large ‖ẑ − z‖, where ẑ is the hidden activations recovered by IHT, z is the true activation). Compared to Algorithm 2, (one-step) IHT is not so robust to high coherence.
In summary, when the assumption of i.i.d Gaussian randomness of the CNN filters holds, our theoretical reconstruction bound strictly match with the empirical observations. More importantly, we demonstrate that the bound can still reasonably hold in practice for discriminatively learned CNN layers, which is particularly true for layers with relatively lower coherence.
5 CONCLUSION
We introduce three concepts that tie together a particular model of compressive sensing (and the associated recovery algorithms), the properties of learned filters, and the empirical observation
13The relative error in activation space of random activations (the last column) are identical (1.414) for all layers because ‖f − f̂‖/‖f‖ = √ 2 on average for Gaussian random f̂ provided ‖f‖ = ‖f̂‖.
that CNNs are (approximately) invertible. Our experiments show that filters in trained CNNs are consistent with the mathematical properties we present while the hidden units exhibit a much richer structure than mathematical analysis suggests. Perhaps simply moving towards a compressive, rather than exactly sparse, model for the hidden units will capture the sophisticated structure in these layers of a CNN or, perhaps, we need a more sophisticated model. Our experiments also demonstrate that there is considerable information captured in the switch units (or the identities of the non-zeros in the hidden units after pooling) that no mathematical model has yet expressed or explored thoroughly.
A MATHEMATICAL ANALYSIS: MODEL-RIP AND RANDOM FILTERS
Theorem 3.1(Restated) Assume that we have MK vectors wi,m of length ` in which each entry is a scaled i.i.d. (sub-)Gaussian random variable with mean zero and variance 1 (the scaling factor is 1/ √ M`). Let t be the stride length (where n = (D − `)/t+ 1) and build the structured random matrix W as the weight matrix in a single layer CNN for M -channel input dimension D. If
M`2
D ≥ Ck
δ2k
( log(K) + log(n)− log( ) ) ,
then, with probability 1− , the MD ×Kn matrix W T satisfies the model-RIP for modelMk with parameter δk.
Proof. We note that this result follows the same structure of that for many proofs of the RIP for (structured) random matrices (see Park et al. (2011); Vershynin (2010) for details) although we make minor tweaks to account for the particular structure of W T .
Suppose that z ∈Mk which means that z consists of at most k non-zero entries that each appear in a distinct block of size n (there are a total of K blocks). First, we observe that the norm of W Tz is preserved in expectation.
Lemma A.1. E(‖W Tz‖22) = ‖z‖22
Proof. Note that each entry of W T is either zero or Gaussian random variable w ∼ N(0, 1) (suitably normalized). Therefore, it is obvious that E(WW T ) = I since each row of W satisfies E ((
wj1i,m1 )T ( wj2i,m2 )) = 0 if j1 6= j2 or m1 6= m2, and we normalized the random variables so
that E (∥∥∥∥[(wji,1)T , . . . ,(wji,M)T]∥∥∥∥
2
) = 1 for all j. Finally, we have
E ( ‖W Tz‖22 ) = E ( zTWW Tz ) = zTE ( WW T ) z = zTz = ‖z‖22.
Let y = W Tz. We aim to show that the square norm of the random variable ‖y‖22 concentrates tightly about its mean; i.e., with exceedingly low probability∣∣∣‖y‖22 − ‖z‖22∣∣∣ > δ‖z‖22. To do so, we need several properties of sub-Gaussian and sub-exponential random variables. A mean-zero sub-Gaussian random variable Z has a moment generating function that satisfies
E(exp(tZ)) ≤ exp(t2C2) for all t ∈ R and some constant C. The sub-Gaussian norm of Z, denoted ‖Z‖ψ2 is
‖Z‖ψ2 = sup p≥1 1 √ p
( E|Z|p )1/p .
If Z ∼ N(0, σ2), then ‖Z‖ψ2 ≤ cσ.
A sub-exponential random variable X satisfies14 P ( |X| > t ) ≤ exp(1− t/C)
14There are two other equivalent properties. See Vershynin (2010) for details.
for all t ≥ 0. Let yi denote the ith entry of the vector y = W Tz. We can write
yi = Kn∑ j=1 W Ti,jzj
and observe that yi is a linear combination of i.i.d. sub-Gaussian random variables (or it is identically equal to 0) and, as such, is itself a sub-Gaussian random variable with mean zero and sub-Gaussian norm ‖yi‖ψ2 ≤ C/ √ M`‖w‖ψ2‖z‖2 (see Vershynin (2010), Lemma 5.9). The structure of the random matrix and how many non-zero entries are in row i of W do enter the more refined bound on the sub-Gaussian norm of ‖yi‖ψ2 (again, see Vershynin (2010), Lemma 5.9 for details) but we ignore such details for this estimate as they are not necessary for the next estimate.
To obtain a concentration bound for ‖yi‖22, we recall from Park et al. (2011); Vershynin (2010) that the sum of squares of sub-Gaussian random variables tightly concentrate.
Theorem A.2. Let Y1, . . . , YMD be independent sub-Gaussian random variables with sub-Gaussian norms ‖Yi‖ψ2 for all i = 1, . . . ,MD. Let T = maxi ‖Yi‖ψ2 . The for every t ≥ 0 and every a ∈ RMD,
P (∣∣∣MD∑ i=1 ai(Yi − EY 2i ) ∣∣∣ ≥ t) ≤ 2 exp(− C min( Ct2 T 2‖a‖22 , Ct T‖a‖∞ )) .
We note that although some entries yi may be identically zero, depending on the sparsity pattern of z, not all entries are. Let us define ỹi = yi‖yi‖ψ2 so that ‖ỹi‖ψ2 = 1 and observe that
P (∣∣∣‖y‖22 − ‖z‖22∣∣∣ > δ‖z‖22) = P (∣∣∣MD∑ i=1 ‖yi‖2ψ2(ỹ 2 i − Eỹ2i ) ∣∣∣ > δ‖z‖22 ) .
We apply Theorem A.2 to the sub-Gaussian random variables ỹi with the weights ‖yi‖2ψ2 . We have
‖a‖22 = MD∑ i=1 ‖yi‖4ψ2 ≤ CD‖w‖4ψ2‖z‖ 4 2 M`2 and ‖a‖∞ ≤ C‖w‖2ψ2‖z‖ 2 2 M` .
If we set T = 1, t = δ‖z‖22, and use the above estimates for the norms of a, we have P (∣∣∣‖y‖22 − ‖z‖22∣∣∣ > δ‖z‖22) ≤ 2 exp ( − C min (Cδ2M`2 D‖w‖4ψ2 , CδM` ‖w‖2ψ2 )) . (4)
Finally, we use the concentration of measure result in a crude union bound to bound the failure probability over all vectors z ∈Mk. We take nk ( K k ) ≈ (nK)k and for a desired constant failure probability. Using the smaller term in Equation (4), (note that δ < 1, `/D < 1, and ‖w‖ψ2 ≥ 1) we have
exp ( − C M` 2δ2
D‖w‖4ψ2
) exp ( k(log(K) + log(n)) ) ≤ exp(log( ))
which implies
M`2
D ≥ k δ2 ‖w‖4ψ2
( log(K) + log(n)− log( ) ) = C k
δ2
( log(K) + log(n)− log( ) ) .
Therefore, if design our matrix W as described and with the parameter relationship as above, the matrix W T with satisfy the model-RIP forMk and parameter δ with probability 1− .
Let us discuss the relationship amongst the parameters in our result. First, if we have only one channel M = 1 and the filter length ` = D, then our bound on the number of measurements D matches those of traditional (model-based) compressive sensing; namely,
D ≥ C k δ2
(log(K) + log(n)− log( )) . If ` < D (i.e., the filters are much shorter than the length of the input signal as in a CNN), then we can compensate by adding more channels; i.e., the filter length ` needs to be larger than √ D, or, if
add more channels, √ D/M .
B MATHEMATICAL ANALYSIS: RECONSTRUCTION BOUNDS
The consequences of having model-RIP are two-fold. The first is that if we assume that an input image is the structured sparse linear combination of filters, x = W Tz (where z ∈ Mk and W T satisfies the model-RIP property), then we know an upper and lower bound on the norm of x in terms of the norm of its sparse coefficients, ‖x‖2 ≤ (1± δ)‖z‖2. Additionally,
‖z‖2 ≤ 1√
1− δ ‖x‖2.
More importantly, when we calculate the hidden units of x,
h = ReLU(Wx) = ReLU(WW Tz)
we can see that the computation of h is nothing other than the first step of a reconstruction algorithm analogous to that of model-based compressed sensing. As a result, we have a bound on the error between h and z and we see that we can analyze the approximation properties of a feedfoward CNN and its linear reconstruction algorithm. In particular, we can conclude that a feedforward CNN and a linear reconstruction algorithm provide a good approximation to the original input image.
Theorem 3.3(Restated) We assume that W T satisfies theM2k-RIP with constant δk ≤ δ2k < 1. If we use W in a single layer CNN both to compute the hidden units ẑ and to reconstruct the input x from these hidden units as x̂ so that x̂ = W TM(Wx, k), the error in our reconstruction is
‖x̂− x‖2 ≤ 5δ2k
1− δk √ 1 + δ2k√ 1− δ2k ‖x‖2.
Proof. To show this result, we recall the two following lemmas from Baraniuk et al. (2010) and rephrase them in the setting of a feedforward CNN.
Lemma B.1. Suppose W T hasMk-RIP with constant δk. Let Ω be a support corresponding to a subspace inMk. Then we have the following bounds:
‖WΩx‖2 ≤ √
1 + δk‖x‖2 (5) ‖WΩW TΩ z‖2 ≤ (1 + δk)‖z‖2 (6) ‖WΩW TΩ z‖2 ≥ (1− δk)‖z‖2 (7)
Lemma B.2. Suppose that W T hasM2k-RIP with constant δ2k. Let Ω be a support corresponding to a subspace ofMk and suppose that z ∈Mk (not necessarily supported on Ω). Then
‖WΩW Tz|Ωc‖2 ≤ δ2k‖z|Ωc‖2.
Let Π denote the support of theMk sparse vector z. Set h = Wx and set ẑ to be the result of max pooling applied to the vector h, or the best fit (with respect to the `2 norm) to h in the modelMk. Let Ω denote the support set of ẑ ∈Mk. For simplicity, we assume |Π| = k = |Ω|. Lemma B.3 (Identification). The support set, Ω, of the switch units captures a significant fraction of the total energy in the coefficient vector z
‖z|Ωc‖2 ≤ 2δ2k
1− δk ‖z‖2.
Proof. Let hΩ and hΠ be the vector h restricted to the support sets Ω and Π, respectively. Since both are support sets forMk and since Ω is the best support set for h,
‖h− hΩ‖2 ≤ ‖h− hΠ‖2, and, after several calculations, we have
‖h|Ω\Π‖22 ≥ ‖h|Π\Ω‖22.
Using Lemma B.2 and the size |(Ω \Π) ⋃ Π| ≤ 2k, we have
‖hΩ\Π‖2 = ‖WΩ\ΠW Tz‖2 ≤ δ2k‖z‖2.
We can bound the other side of the inequality as
‖hΠ\Ω‖2 ≥ ‖WΠ\Ω(W Tz|Π\Ω)‖2 − ‖WΠ\Ω(W Tz|Ω)‖2 ≥ (1− δk)‖z|Π\Ω‖2 − δ2k‖z|Ω‖2.
Since the support of z is the set Π, Π \ Ω = Ωc and we can conclude that
δ2k‖z‖2 ≥ (1− δk)‖z|Ωc‖2 − δ2k‖z|Ω‖2,
and with some rearrangement, we have
‖z|Ωc‖2 ≤ 2δ2k
1− δk ‖z‖2.
To set the value of ẑ on its support set Ω, we simply set ẑ = h|Ω and ẑ|Ωc = 0. Then Lemma B.4 (Estimation).
‖z − ẑ‖2 ≤ 5δ2k
1− δk ‖z‖2
Proof. First, note that ‖I −WΩW TΩ ‖2 ≤ δk since
(1− δk) ≤ sup ‖z‖6=0 ‖W TΩ z‖22 ‖z‖22
( = σ2max(W T Ω ) = σmax(WΩW T Ω ) ) ≤ (1 + δk),
where σmax is the maximum singular value. Therefore,
‖z − ẑ‖2 ≤ ‖z|Ωc‖2 + ‖z|Ω − ẑ|Ω‖2 = ‖z|Ωc‖2 + ‖z|Ω −WΩ(W Tz|Ω + W Tz|Ωc)‖2 ≤ ‖z|Ωc‖2 + ‖(I −WΩW TΩ )z|Ω‖2 + ‖WΩW Tz|Ωc‖2 ≤ ‖z|Ωc‖2 + ‖I −WΩW TΩ ‖2‖z|Ω‖2 + δ2k‖z|Ωc‖2 ≤ ‖z|Ωc‖2 + δk‖z|Ω‖2 + δ2k‖z|Ωc‖2
≤ ( (1 + δ2k) 2δ2k
1− δk + δk
) ‖z‖2
≤ 5δ2k 1− δk ‖z‖2.
Finally, if we use the autoencoder formulation to reconstruct the original image x by setting x̂ = W T ẑ, we can estimate the reconstruction error. We note that ẑ isMk-sparse by construction and remind the reader that W T satisfiesM2k-model-RIP with constants δk ≤ δ2k 1. Then, using Lemma B.4 as well as theM2k-sparse properties of W T ,
‖x− x̂‖2 = ‖W T (z − ẑ)‖2 ≤ √ 1 + δ2k‖z − ẑ‖2
≤ 5δ2k 1− δk
√ 1 + δ2k‖z‖2
≤ 5δ2k 1− δk √ 1 + δ2k√ 1− δ2k ‖x‖2.
This proves that a feedforward CNN with a linear reconstruction algorithm is an approximate autoencoder and bounds the reconstruction error of the input image in terms of the geometric properties of the filters.
C MORE EXPERIMENTAL RESULTS
C.1 MORE DETAILS ON EVALUATION OF CNNS WITH GAUSSIAN RANDOM FILTERS
In this section, we provide more details on the network architectures that we used in Table 1. In particular, we describe the best performing architectures for all cases in Table 5.
C.2 LAYER-WISE COHERENCE AND SPARSITY FOR ALEXNET
We present coherence (see Table 6) and sparsity level (see Table 7) for each layer in AlexNet.
C.3 VISUALIZATION OF IMAGE RECONSTRUCTION FOR VGGNET
In Figure 5, we show reconstructed images from each layer using different reconstruction methods via a pretrained decoding network.

The authors propose a theoretical framework to analyze the recoverability of sparse activations in intermediate layers of deep networks, using theoretical tools from compressed sensing. They relate the computations that are performed by a CNN and a particular recovery algorithm (Iterative Hard Thresholding, IHT). They present proofs of necessary conditions for recoverability to hold, and also show detailed empirical evidence of how they hold in practice.
This is a well-written paper that presents a new angle on why the current CNN architectures work so well. The authors give a brief but sufficient review of the fundamentals of compressed sensing, present their main result relating feed-forward networks and IHT (a surprising result), and progress naturally to a detailed experimental section. The introductory analysis at the beginning of Section 3, in particular, delivers the gist of why the method should work with very approachable and simple math, which is not common in theoretical papers. The increasing complexity of the experiments, done in small steps, shows a nice progression from artificial distributions to a realistic experiment.
A few aspects should be improved. First of all, although the treatment of ReLU non-linearities is sufficient, it is assumed with little discussion that max-pooling non-linearities shouldn't present a problem as well. A discussion of how this is inverted (e.g., with pooling switches) is needed.
The relationship between feed-forward nets and Algorithm 1 assumes tied weights. It might be worthwhile to mention that the result is stronger for the case of RNNs, where this is the case by design.
Although it might be obvious, it might help some readers to briefly note that the reconstruction algorithm is meant to be applied to each layer sequentially, basing the activations of each layer on the one above it (in back-propagation order).
Finally, the filter coherence measure must be defined either mathematically or with a proper reference.