We introduce a sparse scattering deep convolutional neural network, which provides a simple model to analyze properties of deep representation learning for classification. Learning a single dictionary matrix with a classifier yields a higher classification accuracy than AlexNet over the ImageNet 2012 dataset. The network first applies a scattering transform that linearizes variabilities due to geometric transformations such as translations and small deformations. A sparse 1 dictionary coding reduces intra-class variability while preserving class separation through projections over unions of linear spaces. It is implemented in a deep convolutional network with a homotopy algorithm having an exponential convergence. A convergence proof is given in a general framework that includes ALISTA. Classification results are analyzed on ImageNet.
The work considers a new architecture for artificial neural networks to be used in image classification tasks. The network combines several promising previous research directions into an interesting approach (to the best of my knowledge, the architecture is novel). 
In the first step, the input representation (i.e., image) is processed using deep scattering spectrum. This is an operator proposed by Mallat (2012) and it is known for extracting robust features with properties such as translation invariance and Lipschitz continuity of the operator mapping. As a result, such a representation is likely to be more stable to perturbations and noise in the inputs.
The scattering operator of the second and higher orders tends to produce a large number of coefficients. Such a representation is, in general, sparse with many of the higher order coefficients equal to zero. To lower the dimension of the scattering representation, the architecture employs a linear projection operator. For example, one can take principal component analysis and project to a sub-space retaining most of the variation present in the data.
The first two blocks (scattering and linear projection) are unsupervised and, thus, kept fixed during learning. 
The third block in the architecture aims at finding a sparse coding dictionary that will take into account instances and labels. It builds a dictionary using a convolutional neural network and finds sparse coding using previous work on dictionary learning (e.g., Donoho & Elad, 2006: Marial et al., 2011; Jiao et al., 2017 etc). The algorithm works by computing the sparse encoding vector (see Eq. 1) in the forward pass and updating the convolutional parameters as well as sparsity controlling hyperparameter in the backward step. To speed up the convergence, the authors rely on homotopy iterated thresholding illustrated in Figure 2.
The approach is evaluated on ImageNet and empirical results demonstrate that the removal of the sparse encoding block amounts to a significant performance degradation. The results also establish a minor improvement in the accuracy as a result of adding a linear projection matrix (i.e., principal component analysis applied to scattering coefficients). Overall, the network shows promising performance on ImageNet by doing better than AlexNet. The result is not yet 'competitive' with ResNets but it might be worth to pursue this direction of research in the future.
The work is properly structured with well organized materials from previous work. The clarity, however, could be improved in several places. In particular, the last paragraph in Section 3.1 does not explain the algorithm clearly. The confusing part is how to update the dictionary (i.e., convolutional network) parameters. One might infer from the current materials that first the problem in Eq. (1) is solved to find \alpha and that solution is fixed. Then, for that setting of the sparse encoding vector one will train the network parameters. Section 3.2 then givens an iterative procedure in Eq. (3) and Figure 2, which suggest that in a forward pass the dictionary representation is computed using some setting of parameters and \alpha is updated per Eq. (3). Following this, the gradient of the convolutional parameters is computed (\alpha_{n + 1} is differentiated, which means that the gradient depends on other \alpha iterates). I believe that this is the crux of the methodological/conceptual contribution and requires proper explanation with proper illustration of the forward-backward pass.
In Section 3.2 (homotopy iterated thresholding and ALISTA), there is a matrix W which comes ad-hoc. There should be some motivation and gentle introduction. At the moment, it is completely justified to ask why one needs this matrix and why the approach would not work without Proposition 3.1. Please add some discussion and make sure things are properly motivated and gently introduced. This will also place the theoretical contribution in the proper context and strengthen the work.
Just below Figure 2, I fail to follow how the number of layers N relates to the iterative algorithm? Does it mean that you would actually have blocks per each \alpha_n for some N indices (this again refers to previous comment on clarity)?
Can you please use \ell_1 or l_1 notation for sparse dictionary coding. The current symbol reads as 1 to the power of 1 and it is very confusing (never seen it before in the context of sparse encodings).