Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds
In recent years, unfolding iterative algorithms as neural networks has become an empirical success in solving sparse recovery problems. However, its theoretical understanding is still immature, which prevents us from fully utilizing the power of neural networks. In this work, we study unfolded ISTA (Iterative Shrinkage Thresholding Algorithm) for sparse signal recovery. We introduce a weight structure that is necessary for asymptotic convergence to the true sparse signal. With this structure, unfolded ISTA can attain a linear convergence, which is better than the sublinear convergence of ISTA/FISTA in general cases. Furthermore, we propose to incorporate thresholding in the network to perform support selection, which is easy to implement and able to boost the convergence rate both theoretically and empirically. Extensive simulations, including sparse vector recovery and a compressive sensing experiment on real image data, corroborate our theoretical results and demonstrate their practical usefulness. We have made our codes publicly available.
2 .
This paper proves that the unfolded LISTA, an empirical deep learning solver for sparse codes, can guarantee faster (linear) asymptotical convergence than standard iterative ISTA algorithm. The authors proposed a partial weight coupling structure and a support detection schemes. Rooted in standard LASSO techniques, they are both shown to speed up LISTA convergence. The theories are endorsed by extensive simulations and a real-data CS experiment.   Strength: I really like this paper. Simple, elegant, easy-to-implement techniques are backed up by solid theory. Experiments follow a step-by-step manner and accompanies theories fairly well.  - ISTA is generally sub-linearly convergent before its iterates settle on a support. Several prior works [8,15,21] show the acceleration effect of LISTA from different views, but this paper for the first time established the linear convergence of LISTA (upper bound). I view that as an important progress in the research direction of NN sparse solvers. - Both weight coupling and support detection are well motivated by theoretical speedup results. They are also very practical and can be “plug-and-play” with standard LISTA, with considerable improvements observed in experiments  Weakness: I have no particular argument for weakness. Two suggestions for authors to consider: - How the authors see whether their theory can be extended to convolutional sparse coding, which might be more suitable choices for image CS?  - Although IHT and LISTA solve different problems (thus not directly "comparable" in simulation), is it possible that the l0-based networks [4,21] can also achieve competitive performance in real data CS? The authors are suggested to compare in their updated version.