Maximal Sparsity with Deep Networks?
The iterations of many sparse estimation algorithms are comprised of a fixed linear filter cascaded with a thresholding nonlinearity, which collectively resemble a typical neural network layer. Consequently, a lengthy sequence of algorithm iterations can be viewed as a deep network with shared, hand-crafted layer weights. It is therefore quite natural to examine the degree to which a learned network model might act as a viable surrogate for traditional sparse estimation in domains where ample training data is available. While the possibility of a reduced computational budget is readily apparent when a ceiling is imposed on the number of layers, our work primarily focuses on estimation accuracy. In particular, it is well-known that when a signal dictionary has coherent columns, as quantified by a large RIP constant, then most tractable iterative algorithms are unable to find maximally sparse representations. In contrast, we demonstrate both theoretically and empirically the potential for a trained deep network to recover minimal 0 -norm representations in regimes where existing methods fail. The resulting system, which can effectively learn novel iterative sparse estimation algorithms, is deployed on a practical photometric stereo estimation problem, where the goal is to remove sparse outliers that can disrupt the estimation of surface normals from a 3D scene.
This paper investigates the problem of sparse recovery of a linear system with unknown dictionary. Given a set of training signal-response pairs generated from an unknown linear system, the authors introduce a predictive sparse recovery model based on deep neural network (DNN) training. The main idea is to assume that the sparse signal can be encoded by a deep network built over the observation. Inspired by iterative hard-thresholding (IHT), the hard-thresholding operation is applied as activation function on the hidden layers to induce the nonlinearity. Numerical results on synthetic data and photometric stereo applications are reported to show the actual performance of the proposed model. 
The idea of using DNN to approximately unfold the IHT iteration procedure is interesting and novel as far as I am aware of. I have, however, the following major concerns on its content:  (1) Motivation is unconvincing: The training pairs (x, y) are assumed to obey a linear system y=Ax with unknown measurement matrix A. Given ample training data, why should we bother to use DNN-type models if A can simply be estimated via, e.g., least squared regression? Apparently, when A is recovered we may then apply IHT on a testing response vector with the estimated measurement matrix to approximately recover the sparse signal. Actually, I note the simulation study uses a training set of size 600,000 and the measurement matrix is only of size 20x100 << 600,000. In such a setting, it should not be uneasy to accurately estimate A from the data.   (2) Another major concern is its readability. The somewhat overcomplicated writing style in fact underscores the limited innovation of model and algorithmic contributions. Throughout the paper, I cannot find a clear description of the network structure and its optimization algorithm.   (3) There is a clear gap between theory and applications. The main sparse recovery result is about parameter estimation error in a noiseless linear system. The presented application of multi-resolution sparse estimation is defined as a multi-label classification problem to which it remains unclear whether the developed theory can apply.